Embed Size (px)
Citation preview
Event Streaming Pipeline with Windows Azure and ArcGIS GeoEvent Processor
Using Azure EventHub and StreamAnalytics to pre-process a geo-eventstream
Roberto Messora
Value Lab S.p.a., Milan, Italy
ESRI Dev Summit 2015
2
Internet of Things and event streaming are one of the most challenging scenarios in terms of
– Scalability
– High available IT environments
Most of the streamed data are not directly related with geography
In this context managing the entire data stream with ArcGIS GeoEventProcessor could not be the most reliable solution:
–We need to setup and maintain a scalable and high available IT environment
– Not all the data content is meaningful in terms of geo awarness
A better solution could result from the integration between a series of cloud services and ArcGIS GeoEvent Processor
We can build a robust event ingestion and analytics pipeline using some specific Microsoft Azure services as a pre-processor for ArcGIS GeoEventProcessor
Context
3
Azure IoT solution– Architecture and Components
– Event Hub
– Stream Analytics
Azure + ArcGIS GeoEvent Processor solution
Agenda
4
Azure IoT Architecture and Components
Operational Data Stores
Devices
Applications, Websites
Sensors
3rd Party Data Streams
Event Hubs
Blob Storage
Stream Analytics
Machine Learning
HDInsightStorm
Custom Application
Excel (Power Query)
Power BI
Event Hubs
SQL Database
Blob Storage
Data Sources Data Integration Real-Time Analytics Data Storage Compsuntion
5
Event Hub is a highly scalable publish-subscribe ingestor that can intake millions of events per second so that you can process and analyze the massive amounts of data produced by your connected devices and applications.
– Log millions of events per second in near real time
– Connect devices with flexible authorization and throttling
– Time-based event buffering
–Managed service with elastic scale
– Broad platform reach with native client libraries
– Pluggable adapters for other cloud services
Azure Event Hub
6
Stream Analytics is an event processing engine that helps uncover real-time insights from devices, sensors, infrastructure, applications and data.
– Stream millions of events per second
– Perform real-time analytics
– Correlate across multiple streams of data
– Reliable performance and predictable results
– No hardware to deploy
– Rapid development with familiar SQL-like language
Azure Stream Analytics
7
Azure IoT solution
– Architecture and Components
– Event Hub
– Stream Analytics
Azure + ArcGIS GeoEvent Processor solution
Agenda
8
Azure + ArcGIS GeoEvent Processor solution
Devices
Input Event Hub
Stream Analytics
Output Event Hub
Custom .NET Event Processor
(Azure Worker Role)
ArcGIS GeoEventProcessor
Azure
REST JSON
Windowing
Filtering Projections
9
WITH q1 AS (
SELECT deviceId, MAX(eventTimestamp) AS maxTime
FROM Input TIMESTAMP BY eventTimestamp
WHERE temperature >= 28
GROUP BY deviceId, TumblingWindow(second, 10)
)
SELECT q2.deviceId, q2.id, q2.eventTimestamp, q2.temperature, q2.latitude, q2.longitude
FROM Input q2 TIMESTAMP BY eventTimestamp
JOIN q1
ON q2.deviceId = q1.deviceId
AND q2.eventTimestamp = q1.maxTime
AND DATEDIFF(minute, q1, q2) BETWEEN 0 AND 1
Azure Stream Analytics query
{
"deviceId":1,
"id":635616240398378866,
"eventTimestamp":"2015-03-10T22:40:39.8378866-07:00",
"temperature":26,
"latitude":45.0,
"longitude":9.0
}Windowing
Filtering Projections
10
Summary and next steps
Using Microsoft Azure event ingestion and analytics capabilities can be a very efficient solution when dealing with data that contains some sort of geographical reference:
– It’s plug & play
– It’s easily scalable
– It’s very flexible in terms of building blocks composition and communication
protocols
– Can reduce the amount of data streamed to ArcGIS GeoEvent Processor
– You pay only for what you use
Next versions of ArcGIS GeoEvent Processor will introduce a better support for scalability and high available environments, also new cloud Deployment Builder will help to manage ArcGIS architecture, but will not avoid IT issues such as sw/hw maintenance
11
References
Roberto Messora
GIS Specialist Value Lab S.p.a.,Piazza Diaz 220123, MilanoItaly
(+39) 331.1868848
Value Lab SrlPiazza Diaz 2, 20123 Milano (Italy)
T +39.02.7788931 F [email protected] - www.valuelab.it
