IoT Innovation Lab Berlin @relayr - Kay Lerch on Getting basics right for your IoT business...

www.immobilienscout24.de

Get the basics right for your IoT businessIoT Innovation Lab @ Relayr – 2016-05-04 – Speaker: @KayLerch

Agenda

1.Know the tech stack2.AWS can manage things for you3.AWS is not the only truth4.That’s Apache NiFi5.AWS IoT basic concepts (Recap)6.AWS IoT & Apache NiFi7.Focus on business value8.Focus on innovation

Seite 2Get the basics right for your IoT business | Kay Lerch

Know the tech stackbefore you do business

Seite 3Get the basics right for your IoT business | Kay Lerch

A reference architecture for IoT

Seite 4Source: WSO2 (IoT Whitepaper)

Value Value

Value

Value

Effort

Effort

Effort

Challenges:• Set the basics right (EFFORT)• Integrate (VALUE) in your current business

/ IT landscape• Be innovative!

Solution:Minimize (EFFORT)• Rely on managed services and toolsMaximize (VALUE)• Have the basics right• rely on established tools• Think over your integration strategyDrive innovation• Think beyond borders• disrupt business with combining

established things in a totally new fashion

Get the basics right for your IoT business | Kay Lerch

AWS can manage those basicswith a managed IoT solution

Seite 5Get the basics right for your IoT business | Kay Lerch

What AWS (IoT) provides …

Seite 6

Device Registry

TLS Certificates with associated IAM-like policies

Communication over HTTP and MQTT

Message broker with topicsState persistence with shadows

Event processing and forwarding with IoT Rules and Rule actions

Device SDKs in Node.JS, Cand for Arduino Yún

Get the basics right for your IoT business | Kay Lerch

What AWS (IoT) provides …

Seite 7

Event processing and analytics with Kinesis Eventstreams, ML, EMR and Lambda

Data / event stores like S3 or Dynamo DB

API Gateway for public endpoint management

Have your IoT related key metrics in CloudWatch

EC2 as a ground for your (web) applications communicating with devices

Get the basics right for your IoT business | Kay Lerch

AWS IoT core tents

Seite 8

Read this: “Core tenets of IoT” by AWS (as of April 2016)

Get the basics right for your IoT business | Kay Lerch

AWS is not the only truthYou surely have your lovely tools outside AWS

Seite 9Get the basics right for your IoT business | Kay Lerch

A brief overview of data processing and analysisThat‘s quite a lot of tooling and technology …

Seite 10

Hidden complexities

Data Producers

Data Consumers

EventBroker

(Big) data stores

Data analysis challenges (Realtime) Data Processing & AnalyticsIntegrationchallengesIn

gest

ion

chal

leng

es

Data security challenges

Get the basics right for your IoT business | Kay Lerch

A brief overview of data processing and analysisThat‘s quite a lot of tooling and technology …

Seite 11

Hidden complexities

Data Producers

Data Consumers

EventBroker

(Big) data stores

Data analysis challenges (Realtime) Data Processing & AnalyticsIntegrationchallengesIn

gest

ion

chal

leng

es

Data security challenges

Get the basics right for your IoT business | Kay Lerch

If you want …

a (realtime) big picture of your dataflows an option to overlook lineage of each data element have the flexibility to change things on the fly prioritize data overcome challenges of integrating the variety of

technologies with one overarching solution enforce security and compliance along dataflows rely on extensibility driven by OS community satisfy all those needs and keep your tools get rid of only those tools focused on moving data without

making concessions to overall performance

… then you might love:

Just to let you know …

Seite 12Source: WSO2 (IoT Whitepaper)

Value Value

Value

Value

Effort

Effort

Effort

We are here

And a bit of here

Get the basics right for your IoT business | Kay Lerch

That’s Apache NiFi

Seite 13Get the basics right for your IoT business | Kay Lerch

Seite 14

That‘s Apache NiFiin one page

Apache nifi supports powerful and scalable directed graphs of datarouting, transformation, and system mediation logic.Web-based user interfaceSeamless experience between design, control, feedback, and monitoringHighly configurableLoss tolerant vs guaranteed delivery, Low latency vs high throughput, Dynamic prioritization, Flow can be modified at runtime, Back pressureData ProvenanceTrack dataflow from beginning to endDesigned for extensionBuild your own processors and more, Enables rapid development and effective testingSecureSSL, SSH, HTTPS, encrypted content, etc..., Pluggable role-based authentication/authorization

Source: https://nifi.apache.org/ Get the basics right for your IoT business | Kay Lerch

Seite 15

That‘s Apache NiFiin real and feel

Go to NiFi’s interface and understand:

Processors

Templates

Concept of back pressure

Concept of data prioritization

Provenence Graph

Get the basics right for your IoT business | Kay Lerch

NiFi Cluster

NiFi Cluster Manager (NCM)

JVM

Node (Primary)

NiFi Clustered Architecture

JVM

WebserverProvenanceRepository

ContentRepository

FlowfileRepository

REST-APIAdminUI

Webserver REST-APIAdmin UI

Flow Controller

Cluster Manager

Processor 1Processor 2

Isolated Processor

Controller Service 1Controller Service 2Controller Service n

Hear

tbea

t

Lead

er

elec

tion

Repo

rt ch

ange

Embedded Apache Zookeeper

Node (Slave)JVM

WebserverProvenanceRepository

ContentRepository

FlowfileRepository

REST-APIAdminUI

Flow ControllerProcessor 1Processor 2

Isolated Processor

Controller Service 1Controller Service 2Controller Service n

Hear

tbea

t

Repo

rt ch

ange

Sync

St

ate

Sync

St

ate

AWS IoTBasics (Recap)

Seite 17Get the basics right for your IoT business | Kay Lerch

Seite 18

AWS IoTThe Shadow

AWS IoT

Thing

Thing Shadow

Rule

Reports State Mirrors State in Shadow

Gets reported state or sets desired state

Propagates desired state

Receives desired state

Fulfi

lls

desir

ed st

ate

Subscribes to particular messages

AWS Services

Some AWS Resource

Routesmessage

TLS1.2

TLS 1.2 Policy

Get the basics right for your IoT business | Kay Lerch

AWS IoTMQTT topics

AWS IoT

Thing Shadow

get

get/accepted

get/rejected

Request state

Get shadow state

Get error

update

update/accepted

update/rejected

update/delta

Update state

Confirmation

Get error

Changedstate

1

2

1

2

3

Thing topics name pattern: $aws/things/thing_name/...

Get the basics right for your IoT business | Kay Lerch

Apache NiFi & AWS IoTNew processors

Seite 20Get the basics right for your IoT business | Kay Lerch

Seite 21

Apache NiFi & AWS IoTGetIOTMqtt – a MQTT client

AWS IoT

Thing Shadow

updateUpdate state

Establish Connection

Subscribe

Receive state

1

2

3

Flowfile

Get the basics right for your IoT business | Kay Lerch

Seite 22

Apache NiFi & AWS IoTGetIOTMqtt – Configuration

Get the basics right for your IoT business | Kay Lerch

Seite 23

Apache NiFi & AWS IoTGetIOTMqtt – Live demo

Get the basics right for your IoT business | Kay Lerch

Seite 24

Apache NiFi & AWS IoTGetIOTMqtt – Live demo

Get the basics right for your IoT business | Kay Lerch

Seite 25

Apache NiFi & AWS IoTPutIOTMqtt – instruct a „thing“ (but bypass the shadow)

AWS IoT

Thing Shadow

update / delta

Update state

Establish Connection

Publish state

1

2 Flowfile

Flowfile

Get the basics right for your IoT business | Kay Lerch

Seite 26

Apache NiFi & AWS IoTPutIOTMqtt – Configuration

Get the basics right for your IoT business | Kay Lerch

Seite 27

Apache NiFi & AWS IoTPutIOTMqtt – Live demo

Get the basics right for your IoT business | Kay Lerch

Seite 28

Apache NiFi & AWS IoTGetIOTShadow – constantly check last reported state

AWS IoT

Thing Shadow

updateReport state

RequestShadow

Flowfile

Flowfile

Get the basics right for your IoT business | Kay Lerch

Seite 29

Apache NiFi & AWS IoTGetIOTShadow – Configuration

Get the basics right for your IoT business | Kay Lerch

Seite 30

Apache NiFi & AWS IoTPutIOTShadow – instruct a „thing“ over its shadow

AWS IoT

Thing Shadow

update / delta

Desire state

UpdateShadow

Flowfile

Flowfile

Get the basics right for your IoT business | Kay Lerch

Seite 31

Apache NiFi & AWS IoTPutIOTShadow – Configuration

Get the basics right for your IoT business | Kay Lerch

We got the basicsLet’s focus on business value

Seite 32Get the basics right for your IoT business | Kay Lerch

Homework almost done

Seite 33Source: WSO2 (IoT Whitepaper)

Value Value

Value

Value

Effort

Effort

Effort

Challenges:• Set the basics right (EFFORT)• Integrate (VALUE) in your current business• Be innovative!

Solution:Minimize (EFFORT) Rely on managed services and toolsMaximize (VALUE) Have the basics right rely on established tools• Think over your integration strategyDrive innovation• Think beyond borders• disrupt business with combining

established things in a totally new fashion

Get the basics right for your IoT business | Kay Lerch

Agile integration strategy with AWS Simple Workflows at ImmobilienScout

Legacy / 3rd Party

AWS Simple Workflows (SWF)

AWS

SNS

AWSLambda

AWS

S3

AWS

EC2

AWS

ECS

OnPr

emise

Brok

er

Start Workflow

Put File

Bucket Topic

PublishMessage

Start Workflow

ActivityTask

Function

Function

Trusted Applications / Mobile Apps AWS Flow Framework

Start Workflow

EventTrigger

Send DataInvoke Service

DeciderAppsAW

S Cl

oudw

atc

h

AWS

Clou

dTra

il Logs

Metrics

Alarms

Log Measure

Activity Task

Activity Task

Decision Task

Decision Task

Workflow History

AWS Flow Framework

AWS

***

Any

ActivityTask

DecisionTask

Start Workflow

Trigger Trigger

Notify

APICall

Discover

Decision Task

ServiceApps

AWS Flow Framework

ActivityTask

AWS

IAM

Policies

Manage Permissions

Function

LambdaTask

APICall

IoT Device

AWS

IoT

Topic

PublishMessage

Function

Start Workflow

Rule

Dyna

mo

Start Workflow

Put File

Table

Function

Trigger

Partner

AWS

API

Gate

way

StartWorkflow

Invoke

Function

Route

Service

Lambda Task Decision

Task

Hybrid integration platform aiming for Self-Service integration

We got the basicsLet’s focus on innovation

Seite 36Get the basics right for your IoT business | Kay Lerch

Homework done!

Seite 37Source: WSO2 (IoT Whitepaper)

Value Value

Value

Value

Effort

Effort

Effort

Challenges:• Set the basics right (EFFORT)• Integrate (VALUE) in your current business• Be innovative!

Solution:Minimize (EFFORT) Rely on managed services and toolsMaximize (VALUE) Have the basics right rely on established tools Think over your integration strategyDrive innovation• Think beyond borders• disrupt business with combining

established things in a totally new fashion

Get the basics right for your IoT business | Kay Lerch

Gain real business value

Seite 38Source: WSO2 (IoT Whitepaper)

Value Value

Value

Value

Effort

Effort

Effort

1) Combine values of different worlds to find a breathtaking use case

2) Rely on your basics

3) Build a product around your solution

4) Change the world

Value Value+

Effort Effort Effort+ +

+

=

+

Adopt new things…

Get the basics right for your IoT business | Kay Lerch

Gain valueDemo (Smart home)

Seite 39

+ +

Get the basics right for your IoT business | Kay Lerch

Pick some fancy frontend (VALUE) Pick some fancy “backend” (VALUE)

Rely on your basics (EFFORT)

= HaveSmart Home

Gain valueDemo (VoiceOps)

Seite 40

+ +

Get the basics right for your IoT business | Kay Lerch

Pick some fancy frontend (VALUE) Pick some fancy “backend” (VALUE)

Rely on your basics (EFFORT)

= HaveVoiceOps

Once you got the basics right, it’s up to you to …

Seite 41Get the basics right for your IoT business | Kay Lerch

www.immobilienscout24.de

Thanks for you attention. Any questions?Contact:Immobilien Scout GmbHAndreasstraße 1010243 Berlin

Kay LerchFon +49 30 24 301-1149

kay.lerch@immobilienscout24.de

Backup

Seite 43Get the basics right for your IoT business | Kay Lerch

T-Cobbler

I need some “thing” …

Input & Output units on your breadboard

I need some “thing” …

Control GPIO-Pins with sending or receiving electric impulses

Control an output Control an input

Communicating with AWS IoT over MQTT topics

Alexa Skill AWS Lambda

AWS API Gateway

LambdaPost

Message to Slack

LambdaUpdate Shadow

Rpi 2

AWS IoT

Thing Shadow

Python ScriptGPIO

Rule

Invoke

MQTTPublish

MQTT Receive

Update Shadow on state change

Impulse Sequence

SVC

OutgoingWebhook

Post Message over Slack Web API

433 Mhz Signal

C++ D/AConverter

Post Message if no state changes

AWS Lambda

AWS EC2

LambdaStart / Stop

EC2-Instance

Rpi 2

AWS IoT

Thing Shadow

Python ScriptGPIO

Rule

Invoke

MQTTPublish

Impulse Sequence

Post Message over Slack Web API

433 Mhz Signal

C++ A/DConverter

Start / Stop over AWS SDK

AWS CW

LambdaReact on EC2 state change

Rule

Update Shadow

MQTT Receive

Impulse

On state change

Rpi 2

AWS IoT

Thing Shadow

Python ScriptGPIO

Rule

MQTTPublish

Impulse Sequence

Post Message over Slack Web API

AWS CWAlarmOn Humidity > 30%Metric Metric

Alexa Skill

Read out temperature and humidity