Upload
lei-xu
View
681
Download
1
Embed Size (px)
Citation preview
EVENT DRIVEN STREAMING ANALYTICSDemonstration on Architecture of IoT
Lei Xu 2016.09
1.DEMO SCENARIO
Taxi/Devices
Coordinates & Payment Info
Devices generate transactions after payment
Taxi/Devices
Data FlowCoordinates & Payment Info
IoT Gateway
Applications & Services
Data Management
Cloud Infrastructure
IoT Connectivity Platform
Online Analytics • Dropoff-district Count during the
last 30 min • Real-time Taxi Dropoff Events • 10 Most Frequent Routes(Pickup-
area Dropoff-area) during the last 30 minutes
• 10 Most Profitable areas within the last 15 minutes
Offline Analytics • 10 Most Frequent Routes by week day • 10 Most Profitable Areas by week day • Most Profitable Hours at the 10 Most
Profitable Areas by week day
Transaction being sent to IoT Platform for Analytics
Thousands of Browsers Hundreds of BI Clients
Data ManagementMulti-Channel Data Integration
Structured & Unstructured Data
Analytics & Machine Learning
Cloud InfrastructureCloud Native Platform
Scalability & High Availability
Security & Governance
Visualize DataReporting, BI & User Apps
History Data & Real-time Data
Multiple Devices & Platforms
IoT GatewayDevice Management
Authentication & Authorization
Gateways & Interfaces
Apps & ServicesRule Engine & Process Mgmt.
Complex Event Processing
Business & Data MicroServices
IoT Platform
Components
2.REQUEST DRIVEN FLOW
Storing Enriched Data
Time Based Enrichment &
Filtering
Receiving & Storing Raw Events
Time Based Batch Aggregation
Storing Aggregated Data
Sending Raw Events
Polling Requests for Latest Data
Taxi/Devices
Traditional Request Driven Flow
Event-driven Flow
Request-driven Flow
Time-based Flow
Analytics
BI Visualization
DRAWBACKS & CHALLENGES• Polling + Request/Response Style => Overhead
• Time Based Batch Process => High Latency
• Architecture - Monolith or SOA will work well with HA?
• How to scale & how to maintain system performance with lots of user requests?
• How to scale & how to maintain system performance with huge amount of data?
• Portable for both Cloud and On-Premise environments?
REQUEST DRIVEN & EVENT DRIVENRequest Driven Event Driven
How is an action being taken?
As a response to a specific request
Triggered by the fact of a specific situation
When is an action being taken?
When the request is being processed
Determined by the context of the situation
What happens when the event/request occurs? A response is produced
The event can be ignored, increment the state, trigger an internal derive event, or
trigger a solution
http://epthinking.blogspot.jp/2012/12/more-on-request-driven-vs-event-driven.html
3.EVENT DRIVEN FLOW
Publishing/Subscribing
Enriched Events
Enrichment &
Filtering
Receiving Raw Events
Real-time Aggregation/
Analytics
Publishing/Subscribing
Aggregated Events
Sending Raw Events
Events Visualization
Online AnalyticsTaxi/Devices
Expectations
• Event Driven
• Low Latency
• Scalability
• High Availability
• Portability
• Light Weight
Event-driven Flow
Publishing/Subscribing
Enriched Events
Enrichment &
Filtering
Receiving Raw Events
Real-time Aggregation/
Analytics
Publishing/Subscribing
Aggregated Events
Sending Raw Events
Events Visualization
Online & Offline Analytics
Storing Data for Analytics
Analytics
BI Visualization
Taxi/Devices
Event-driven Flow
Request-driven Flow
Publishing/Subscribing
Enriched Events
Enrichment &
Filtering
Receiving Raw Events
Real-time Aggregation/
Analytics
Publishing/Subscribing
Aggregated Events
Sending Raw Events
Events Visualization
Machine Learning Loopback to Streaming
Storing Data for Analytics
Analytics & Machine Learning
BI Visualization
Taxi/Devices
Event-driven Flow
Request-driven Flow
4.EVENT DRIVEN ARCHITECTURE
Binding Abstraction Layer
Messaging Options
Data Pipeline
Data Microservices
Data
Spring Cloud Stream App
Spring Cloud Stream App
Spring Cloud Stream App
Spring Cloud Stream/Data Flow as Event Driven Architecture
Data
Filter, Enrich, Transform, Aggregation, Rules, CEP, Analytics
Publishing/Subscribing Geocoded Events
Reverse Geocoding
Receiving Raw Events
Real-time Aggregation/Analytics
Publishing/Subscribing Aggregated Events
Sending Raw Events
Events Visualization
Simulator
HTTP Source
MongoDB Processor
GemFire Sink
GemFire Source
WebSocket Sink
WebSocket Sink
Reverse GeocodingReal-time Aggregation/Analytics
Messaging
Apps & Services – Online Analytics
Publishing/Subscribing Geocoded Events
Reverse Geocoding
Receiving Raw Events
Real-time Aggregation/Analytics
Publishing/Subscribing Aggregated Events
Sending Raw Events
Events Visualization
Simulator
HTTP Source
MongoDB Processor
GemFire Sink
GemFire Source
WebSocket Sink
WebSocket Sink
Reverse GeocodingReal-time Aggregation/Analytics
Messaging
BI Visualization
Analytics & Machine Learning
Storing Data for Analytics
HDFS Sink
SQL Query
Apps & Services – Online & Offline Analytics
MPP SQL on HDFS
Publishing/Subscribing Geocoded Events
Reverse Geocoding
Receiving Raw Events
Real-time Aggregation/Analytics
Publishing/Subscribing Aggregated Events
Sending Raw Events
Events Visualization
Simulator
HTTP Source
MongoDB Processor
GemFire Sink
GemFire Source
WebSocket Sink
WebSocket Sink
Reverse GeocodingReal-time Aggregation/Analytics
Messaging
BI Visualization
Analytics & Machine Learning
Storing Data for Analytics
MPP SQL on HDFS
HDFS Sink
SQL QueryMachine Learning
Apps & Services – Machine Learning Loopback to Streaming
Machine Learning
5.ONLINE ANALYTICS LOCAL DEPLOYMENT
MongoDB Processor
Real-time Aggregation/Analytics
Spring Boot App
Spring Cloud Stream App
Spring Cloud Stream App
HTTP Source
GemFire Sink
GemFire Source
WebSocket Sink
Simulator
Spring Cloud Stream App
Spring Cloud Stream App
Spring Cloud Stream App
Java CLI App
Messaging
WebSocket Sink
Spring Cloud Stream App
MVW App
Data Flow
Deployment
DeploymentReverse Geocoding
MongoDB Processor
Real-time Aggregation/Analytics
Spring Boot App
Spring Cloud Stream App
Spring Cloud Stream App
HTTP Source
GemFire Sink
GemFire Source
WebSocket Sink
Simulator
Spring Cloud Stream App
Spring Cloud Stream App
Spring Cloud Stream App
Java CLI App
Messaging
WebSocket Sink
Spring Cloud Stream App
MVW App
Data Flow
Deployment
Scale Out When NecessaryReverse Geocoding
MongoDB Processor
Real-time Aggregation/Analytics
Spring Boot App
Spring Cloud Stream App
Spring Cloud Stream App
HTTP Source
GemFire Sink
GemFire Source
WebSocket Sink
Simulator
Spring Cloud Stream App
Spring Cloud Stream App
Spring Cloud Stream App
Java CLI App
Messaging
WebSocket Sink
Spring Cloud Stream App
MVW App
Data Flow
Deployment
RuntimeReverse Geocoding
6. PRODUCTION LEVEL DEPLOYMENT
MQTT Source
Enrichment Processor
IMDG Sink
Machine Learning
Processor
WebSocket Sink
MPP/HDFS Sink
Real-time IMDG Sink
MPP/HDFS Source
Data Pipelines
IMDG Source
Broker by RabbitMQ
WebSocket by RabbitMQ
Front-end Services
Spring Cloud Stream Apps
MVW App Spring Cloud Stream Apps
Data MicroServicesBusiness MicroServices
……
Containerized Applications
Taxi/Devices
• Event Driven Spring Cloud Stream
• Low Latency Messaging by RabbitMQ Real-time by GemFire
• Scalability Apps Scale-Out by Containers Governance by Data Flow Services Scale-out
• High Availability Platform HA by Cloud Native Platform or PaaS
• Portability Apps & Services portable to other Cloud Native Platforms
• Light Weight Loosely Coupled MicroServices
or or etc.
Back-end ServicesRabbitMQGeode
MongoDB HAWQ
or
Spring Cloud Data Flow
7.DEPLOYMENT & RUNTIME SCREENSHOTS
• Real-time Taxi Dropoff Events
• Dropoff-district Count
• 10 most frequent routes
• 10 most frequent routes
8. LOCAL DEPLOYMENT DEMO VIDEO