View
1.615
Download
0
Category
Preview:
Citation preview
Building a Smarter Home with Apache NiFi and Spark
Joseph Niemiec, Systems ArchitectChristopher Gambino, Solutions Engineer
2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Our Background
Systems Architect– Hadooping for 4+ Years – War-Driving - Mad max style– Telematics Patent in Parking Floor Detection– Co-Author Apache Hadoop YARN Book
Smart Production Lines– Medical Device Quality Control– FDA Compliant Reporting– Custom IOT devices and framework
3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Agenda
Use Case Explanation Hardware & Software Summary Architecture NiFi Development Patterns – Multiple Generations Discoveries Challenges //Other Lessons Learned
4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Use Case Explanation
Hortonworks Data Flow (NiFi) enables easy access and filtering to data that resides on theedge. This data can be transmitted site to site and accessed via data explorationframeworks such as Spark.
Joe was Paranoid Living In Detroit Wanted to investigate the unseen daily activities Understand problems with edge collection and consumption
The Solution?More Technology
5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Software & Hardware Used
Brand Name Smarthome Sensors + Hub– Door, Motion, Window
NiFi on a Raspberry Pi 2 Model B– Raspbian OS – Python Push based collector
NiFi on EC2 EC2 HDP Cluster BareMetal HDP + HDF Pig & Spark
6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Architecture High LevelHub & Spoke Approach
– Hub
• Hadoop Cluster in the DC• Stores All Historical Data• Aimed at making it usable for analysis's• Consumes from Spoke Brokers to land data in DC
– Spoke
• Per Region/Geo• Real Time Stream Processing for Customer Apps• Cleanse data to meet legal requirements for off-geo Hub
7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Architecture Diagram
8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Bidirectional Communication
Edge to Cloud– Collection (Push) Site-2-Site– Updates (Pull) HTTP Calls
Cloud – Distributed Cache of Rules for Edges– HTTP Call Edge Updates– Site-2-Site or HTTP Call Rule Updates
Data Center to Cloud– Collection with Site-2-Site pull from the Cloud to HDFS– HTTP Call Rule Updates
• Easily as GET or POST Cache Updates• Bulk Site-2-Site Push
9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Home Collector High Level
Collection
Monitoring
Redact + Encrypt +Transfer
10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
1st Gen - Blocking, Sequential Writes
Decrypt->Local Disk -> S3 -> Database
Decrypt
To DiskTo S3To RDS
11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
1st Generation Nifi - Separate Channels
Parallel Decryption = Additional Management
Decrypt
Decrypt
12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
1st Gen - Separate Channels
Parallel Writes = Additional Management
To DiskTo Disk
Merge
13 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
2nd Gen - Asynchronous Writes – Still Parallel
Local Disk, S3, and RDS all isolated
Decrypt To Disk
To S3
To RDS
14 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
3rd Generation Nifi - Common Architecture
Add Tag
Decrypt
Add Tag
15 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
3rd Generation Nifi - Merge on Variable
Extremely Low Overhead
Merge To Disk
16 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
3rd Generation Nifi
Routing on Attributes Split Flow
To Disk
17 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
What We Discovered
Joe’s wife absolutely hates being spied on Pre-empt Arrival with MAC Addresses Validate Occupancy based on connected MAC Addresses Hackers Attacking the Network High Level of Control vs. Hub Software Motion almost useless as standalone factor.
18 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Challenges // Other Lessons Learned
Oracle vs. Open JDK for ARM Improperly Configured Backpressure FlowFile Names Site to Site Configuration Issues Execute Process has ambiguous failure modes Merging Files Larger SmartHome Hub interface
19 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Q&ACome hang out with us @ The Dev Café
Recommended