Upload
guido-schmutz
View
574
Download
1
Tags:
Embed Size (px)
Citation preview
2014 © Trivadis
19.11.2014 IoT Architektur
1
IoT Architektur Sind traditionelle Architekturen gut genug, oder braucht es neue Ansätze?
Bernd Trops (Talend) Guido Schmutz (Trivadis) 24.03.2015
Open Source Integration
2014 © Trivadis
Bernd Trops
• Working for Talend for more than 7 years • Co-Author of different books • Consultant, Trainer and Software Architect for Java, SOA • Masons of SOA: http://www.soacommunity.com/index.php/institutional/masons-of-soa • Director Professional Services DACH at Talend • More than 25 years of software development
experience
• Contact: [email protected]
19.11.2014
IoT Architektur 2
2014 © Trivadis
Guido Schmutz
• Working for Trivadis for more than 17 years
• Oracle ACE Director for Fusion Middleware and SOA • Co-Author of different books • Consultant, Trainer Software Architect for Java, Oracle, SOA and
Big Data / Fast Data • Member of Trivadis Architecture Board • Technology Manager @ Trivadis
• More than 25 years of software development experience
• Contact: [email protected] • Blog: http://guidoschmutz.wordpress.com • Twitter: gschmutz
19.11.2014 IoT Architektur
3
2014 © Trivadis
Agenda
1. Introduction / Motivation
2. IoT Reference Architecture
3. Components supporting the IoT Reference Architecture
4. Talend – Unified Platform for IoT and Big Data
5. Summary
19.11.2014 IoT Architektur
4
2014 © Trivadis
Data, Data, Data – Predicted Growth from IoT
19.11.2014 IoT Architektur
5
Source: Brideworks http://bit.ly/19Msx5N
2014 © Trivadis
Data as an Asset - Store Anything?
But then data is just too valuable to delete! We must store anything!
Nonsense! Just store the data
you know you need today!
It depends … but Big Data technologies allow to store the raw information from both new data sources as well as existing ones so that you can later use it to create new data-driven products, you would not have thought about today!
6
19.11.2014 IoT Architektur
2014 © Trivadis
Requirements for an IoT Architecture
• Support for failure • systems should gracefully degrade into lower performing states • If a failed component recovers, it should be able to rejoin the cluster
• Recoverability • In the event of a failure, no data should be lost
• Consistency • Failure of one task should not affect the result of the entire job
• Scalability • Adding load leads to a decline of performance, not failure • Increasing resources results in proportional increase in capacity • Scale-up but also scale-down => elasticity
7
19.11.2014 IoT Architektur
2014 © Trivadis
Agenda
1. Introduction / Motivation
2. IoT Reference Architecture
3. Components supporting the IoT Reference Architecture
4. Talend – Unified Platform for IoT and Big Data
5. Summary
19.11.2014 IoT Architektur
8
2014 © Trivadis
IoT Solution Types
19.11.2014 IoT Architektur
9
Device
On-Device Logic
Gateway
Processing Backend
Enterprise Apps
Enterprise IoT M2M Local Hub Device to Device (P2P)
Device to Backend
Adapted from: http://enterprise-iot.org
2014 © Trivadis
Ente
rpris
e Se
rvic
e Bu
s (E
SB)
Enterprise Applications
WS
External Cloud Service
Providers
BPM and SOA Platform
Event
Business Logic/Rules
Business Intelligence
IoT Device
Services
WS
Event
Processes
Gateway
Event Procesing
IoT Smart Device
Viszualization
Analytics
DB
19.11.2014 IoT Architektur
10
API G
atew
ay
= one way = request/response
1) IoT with Simple Event Processing
10
Mobile Apps
DB
Rich (Web) Client Apps
DB
2014 © Trivadis
Ente
rpris
e Se
rvic
e Bu
s (E
SB)
Stream Processing Infrastructure
Enterprise Applications
WS
External Cloud Service
Providers
BPM and SOA Platform
Event
Business Logic/Rules
Business Intelligence
IoT Device
Ente
rpris
e Ev
ent B
us (I
ngre
ss)
Ente
rpris
e Ev
ent B
us
Services
WS
Event
Processes
Gateway
ESP/CEP
ESP/CEP
IoT Smart Device
Viszualization
Analytics
DB
DB
DB
19.11.2014 IoT Architektur
11
API G
atew
ay
= one way = request/response
11
Mobile Apps
DB
Rich (Web) Client Apps
DB
Social Media
Streams
2) IoT with Event Stream/Complex Event Processing
2014 © Trivadis
Ente
rpris
e Se
rvic
e Bu
s (E
SB)
3) IoT with Big Data Analytics
Enterprise Applications
WS
External Cloud Service
Providers
BPM and SOA Platform
Event
Business Logic/Rules
Business Intelligence
IoT Device
Ente
rpris
e Ev
ent B
us (I
ngre
ss)
Ente
rpris
e Ev
ent B
us
Services
WS
Event
Processes
Gateway
ESP/ CEP
Hadoop Big Data Infrastructure HDFS
Batch Processing
IoT Smart Device
Viszualization
Analytics
DB DB
19.11.2014 IoT Architektur
12
API G
atew
ay
= one way = request/response
12
Mobile Apps
DB
Rich (Web) Client Apps
DB
Social Media
Streams
2014 © Trivadis
Ente
rpris
e Se
rvic
e Bu
s (E
SB)
Stream Processing Infrastructure
Enterprise Applications
WS
External Cloud Service
Providers
BPM and SOA Platform
Event
Business Logic/Rules
Business Intelligence
IoT Device
Ente
rpris
e Ev
ent B
us (I
ngre
ss)
Ente
rpris
e Ev
ent B
us
Services
WS
Event
Processes
Gateway
ESP/CEP
Hadoop Big Data Infrastructure HDFS
ESP/CEP
Batch Processing
IoT Smart Device
Viszualization
Analytics
DB
DB
DB
DB
19.11.2014 IoT Architektur
13
API G
atew
ay
4) IoT with “Lambda” Architecture
= one way = request/response
13
Mobile Apps
DB
Rich (Web) Client Apps
DB
Social Media
Streams
2014 © Trivadis
Ente
rpris
e Se
rvic
e Bu
s (E
SB)
Stream Processing Infrastructure
Enterprise Applications
WS
External Cloud Service
Providers
BPM and SOA Platform
Event
Business Logic/Rules
Business Intelligence
IoT Device
Ente
rpris
e Ev
ent B
us (I
ngre
ss)
Ente
rpris
e Ev
ent B
us
Services
WS
Event
Processes
Gateway
ESP/CEP
Hadoop Big Data Infrastructure HDFS
ESP/CEP
Replay to reprocess
IoT Smart Device
Viszualization
Analytics
DB
DB
DB
19.11.2014 IoT Architektur
14
API G
atew
ay
5) IoT with “Kappa” Architecture
= one way = request/response
14
Mobile Apps
DB
Rich (Web) Client Apps
DB
Social Media
Streams
2014 © Trivadis
Ente
rpris
e Se
rvic
e Bu
s (E
SB)
Stream Processing Infrastructure
Enterprise Applications
WS
External Cloud Service
Providers
BPM and SOA Platform
Event
Business Logic/Rules
Business Intelligence
IoT Device
Ente
rpris
e Ev
ent B
us (I
ngre
ss)
Ente
rpris
e Ev
ent B
us
Services
WS
Event
Processes
Gateway
ESP/CEP
Hadoop Big Data Infrastructure HDFS
ESP/CEP
Batch Processing
IoT Smart Device
Viszualization
Analytics
DB
DB
DB
DB
19.11.2014 IoT Architektur
15
API G
atew
ay
6) IoT with Big Data Advanced Analytics
= one way = request/response
15
Mobile Apps
DB
Rich (Web) Client Apps
DB
Social Media
Streams
2014 © Trivadis
Ente
rpris
e Se
rvic
e Bu
s (E
SB)
Stream Processing Infrastructure
IoT Reference Architecture
Enterprise Applications
WS
External Cloud Service
Providers
BPM and SOA Platform
Event
Business Logic/Rules
Business Intelligence
IoT Device
16
Mobile Apps
DB
Rich (Web) Client Apps
DB
Social Media
Streams
Ente
rpris
e Ev
ent B
us (I
ngre
ss)
Ente
rpris
e Ev
ent B
us
Services
WS
Event
Processes
Gateway
ESP/CEP
Hadoop Big Data Infrastructure HDFS
ESP/CEP
Batch Processing
IoT Smart Device
Viszualization
Analytics
DB
DB
DB
DB
19.11.2014 IoT Architektur
16
API G
atew
ay
= one way = request/response
2014 © Trivadis
IoT Reference Architecture
19.11.2014 IoT Architektur
17 En
terp
rise
Serv
ice
Bus
(ESB
)
Stream Processing Infrastructure
Enterprise Applications
WS
External Cloud Service
Providers
BPM and SOA Platform
Event
Business Logic/Rules
Business Intelligence
IoT Device
17
Mobile Apps
DB
Rich (Web) Client Apps
DB
Social Media
Streams
Ente
rpris
e Ev
ent B
us (I
ngre
ss)
Ente
rpris
e Ev
ent B
us
Services
WS
Event
Processes
Gateway
ESP/CEP
Hadoop Big Data Infrastructure HDFS
ESP/CEP
Batch Processing
IoT Smart Device
Viszualization
Analytics
DB
DB
DB
DB
API G
atew
ay
= one way = request/response
2014 © Trivadis
Ente
rpris
e Se
rvic
e Bu
s (E
SB)
Stream Processing Infrastructure
Enterprise Applications
WS
External Cloud Service
Providers
BPM and SOA Platform
Event
Business Logic/Rules
Business Intelligence
IoT Device
18
Mobile Apps
DB
Rich (Web) Client Apps
DB
Social Media
Streams
Ente
rpris
e Ev
ent B
us (I
ngre
ss)
Ente
rpris
e Ev
ent B
us
Services
WS
Event
Processes
Gateway
ESP/CEP
Hadoop Big Data Infrastructure HDFS
ESP/CEP
Batch Processing
IoT Smart Device
Viszualization
Analytics
DB
DB
DB
DB
API G
atew
ay
19.11.2014 IoT Architektur
18
IoT Reference Architecture – Supporting Components
2014 © Trivadis
Agenda
1. Introduction / Motivation
2. IoT Reference Architecture
3. Components supporting the IoT Reference Architecture
4. Talend – Unified Platform for IoT and Big Data
5. Summary
19.11.2014 IoT Architektur
19
2014 © Trivadis
Enterprise Event Bus - Apache Kafka
• A distributed publish-subscribe messaging system
• Designed for processing of real time activity stream data (logs, metrics collections, social media streams, …)
• Initially developed at LinkedIn, now part of Apache
• Does not follow JMS Standards and does not use JMS API
• Kafka maintains feeds of messages in topics
19.11.2014 IoT Architektur
20
Kafka Cluster
Consumer Consumer Consumer
Producer Producer Producer
0 1 2 3 4 5 6 7 8 9 1 0
1 1
1 2
0 1 2 3 4 5 6 7 8 9
0 1 2 3 4 5 6 7 8 9 1 0
1 1
1 2
Anatomy of a topic:
Partition 0
Partition 1
Partition 2
Writes
old new
2014 © Trivadis
Enterprise Event Bus - Apache Kafka
Kafka at LinkedIn
Up to 2 million writes/sec on 3 cheap machines § Using 3 producers on 3 different machines
19.11.2014 IoT Architektur
21
10+ billion writes per day
172k messages per second
(average)
55+ billion messages per day
to real-time consumers
http://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines
2014 © Trivadis
Management /Monitoring
Infrastructure
Batch (Big) Data Processing - Hadoop Ecosystem
22
Core
Analytics Workflow/Job Unstructured Data Sources
Structured Data Sources
SQL on Hadoop
Serialization Data Storage Security
19.11.2014 IoT Architektur
2014 © Trivadis
Batch (Big) Data Processing - Classical Hadoop vs. Spark
Spark Hadoop Map Reduce
19.11.2014 IoT Architektur
23
HDFS
Map
Map
Reduce
Reduce
Data Data
Map
Join
Transform
Cache
Filter
Join
Data
2014 © Trivadis
Stream Processing – Apache Storm
Tuple • Immutable Set of Key/value pairs
Stream • an unbounded sequence of tuples that can be processed in parallel by Storm
Topology • Wires data and functions via a DAG (directed acyclic graph) • Executes on many machines similar to a MR job in Hadoop
Spout • Source of data streams (tuples) • can be run in “reliable” and “unreliable” mode
Bolt • Consumes 1+ streams and produces new streams • Complex operations often require multiple
steps and thus multiple bolts
19.11.2014 IoT Architektur
24
Spout
Spout
Bolt
Bolt
Bolt
Bolt
Source of Stream B
Subscribes: A Emits: C
Subscribes: A Emits: D
Subscribes: A & B Emits: -
Subscribes: C & D Emits: -
T T T T T T T T
2014 © Trivadis
Data Storage (Batch) – Cloudera Impala
High-performance SQL engine for vast amounts of data
• Massively-parallel processing (MPP)
• Inspired by Google's Dremel project
• Query latency measured in milliseconds
Impala runs on Hadoop clusters
• Can query data stored in HDFS or HBase tables
• Reads and writes data in common Hadoop file formats
Developed by Cloudera
• 100% open source, release under the Apache software license
19.11.2014 IoT Architektur
25
2014 © Trivadis
Data Storage (Speed) – Apache Cassandra
• Developed at Facebook
• Open source distributed database management system
• Professional grade support from company called DataStax
• Main Features § Real-Time § Highly Distributed § Support for Multiple Data Center § Highly Scalable § No Single Point of Failure § Fault Tolerant § Tunable Consistency § Cassandra Query Language (CQL)
19.11.2014 IoT Architektur
26
2014 © Trivadis
Data Storage (Speed) – Apache Cassandra
• Don’t think of relational table => more of a sorted map
• Know your application => model around the queries
• De-normalize and duplicate for read performance
• Index is not an afterthought, anymore=> index upfront
• Think of physical storage structure
19.11.2014 IoT Architektur
27
2013-08 Day-1, keyword-1=>100
Day-2, keyword-1=>150
Day-3, keyword-1=>170 …. Day-31,
keyword-1 =>170
2013-08-31 Hour-1, keyword-1 =>10
Hour-2, keyword-1 =>15
Hour-3, keyword-1 =>17 …. Hour-24,
keyword-1 =>17
2013-08-31-10 Minute-1, keyword-1=>2
Minute-2, keyword-1=>3
Minute-3, keyword-1 =>5 …. Minute-60,
keyword-1=>2
Row-key Columns à
2014 © Trivadis
Agenda
1. Introduction / Motivation
2. IoT Reference Architecture
3. Components supporting the IoT Reference Architecture
4. Talend – Unified Platform for IoT and Big Data
5. Summary
19.11.2014 IoT Architektur
28
2014 © Trivadis
Ente
rpris
e Se
rvic
e Bu
s (E
SB)
Stream Processing Infrastructure
Enterprise Applications
WS
External Cloud Service
Providers
BPM and SOA Platform
Event
Business Logic/Rules
Business Intelligence
IoT Device
29
Mobile Apps
DB
Rich (Web) Client Apps
DB
Social Media
Streams
Ente
rpris
e Ev
ent B
us (I
ngre
ss)
Ente
rpris
e Ev
ent B
us
Services
WS
Event
Processes
Gateway
ESP/CEP
Hadoop Big Data Infrastructure HDFS
ESP/CEP
Batch Processing
IoT Smart Device
Viszualization
Analytics
DB
DB
DB
DB
API G
atew
ay
Talend – Unified Platform for IoT and Big Data
19.11.2014 IoT Architektur
29
2014 © Trivadis
Agenda
1. Introduction / Motivation
2. IoT Reference Architecture
3. Components supporting the IoT Reference Architecture
4. Talend – Unified Platform for IoT and Big Data
5. Summary
19.11.2014 IoT Architektur
32
2014 © Trivadis
Summary
• IoT – The technologies are ready to use
• IoT tends to make Big Data infrastructures necessary
• There are architectures to deal with IoT -> Backend
• Know your use case/requirements to choose the right architecture
• Can my existing backend landscape handle the new IoT load?
• Do I have to handle huge amount of events in “real-time”?
• Do I need to filter/aggregate data before invoking existing backend systems?
• Do I want to do Advanced Analytics (predictive analytics) where large amounts of historical information is necessary?
• Business cases starting to increase
19.11.2014 IoT Architektur
33
2014 © Trivadis
Weitere Informationen...
19.11.2014 IoT Architektur
34
INFOBOX – Lesen und Löschen • Folie wenn auf weitere Informationen
verwiesen werden soll, also z.B. Bücher, Websiten, etc.
2014 © Trivadis
BASEL BERN BRUGG LAUSANNE ZÜRICH DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. HAMBURG MÜNCHEN STUTTGART WIEN
Fragen und Antworten...
2013 © Trivadis
Guido Schmutz
Technology Manager
19.11.2014 IoT Architektur
INFOBOX – Lesen und Löschen • Die Schlussfolie steht in zwei Varianten
zur Verfügung, einmal für die Kontaktdaten eines Referenten, einmal in der Variante für zwei oder mehr Referenten
• Name, Titel und Location jeweils untereinander in eine Zeile (Shift+Return)
• Die Idee ist das diese Folie als letzte Folie (auch für Fragen und Antworten) am Ende der Präsentation lange stehen bleibt, somit haben die Zuhörer die Möglichkeit die Kontaktdaten aufzuschreiben J