ArcGIS GeoEvent Server – OverviewThomas Paschke

1

2

3

4

5

Agenda

GeoEvent Development

GeoEvent Server

Real-Time & Big Data @ 10.5.x

Visualization

Road Ahead (10.6)

R&D Project Trinity & Other6

GeoEvent Development1

GIS DataWhat has happened, what is happening, what will happen

The ‘current’ snapshot is outdated almost as soon as it’s created…

Credit: iStockphoto/chris_lemmens

Real-Time GIS RequirementsIngestion, Analytics, Notifications and Alerting

Requirement #1

Requirement #2

Continuous

AnalysisFeatures

Position

AlertInside

Boundary

Requirement #3

Applications

ArcGIS Serverwith real-time capabilities

ingest visualize

analyze store

real-time & big data

BYO-RDBMSGeoEvent

Server

Web GISwith real-time capabilities

Web GIS

ingest visualize

analyze store

real-time & big data

thousands3K e/s

3K e/s 200 e/s

BYO-RDBMSGeoEvent

Server

X

X

200 e/s

Stream Service

Web GISwith real-time capabilities

Web GIS

ingest visualize

analyze store

real-time & big data

thousands3K e/s

3K e/s 200 e/s

GeoEvent

Server

X

X

200 e/s

relational

data store

Web GISwith real-time capabilities

Web GIS

spatiotemporal

big data store

IoT

ingest visualize

analyze store

real-time & big data

millions4K e/s

4K e/s 10Ks e/s

4K e/s

GeoEvent

Server

GeoEvent Server2

ArcGIS Enterprisewith real-time capabilities

ArcGIS

Enterprise

GeoEvent

Server

IoT

you can create

your own

connectors

Ingestionuse an existing input connector

GeoEvent Server

Inp

uts

Ou

tpu

ts

GeoEvent Services

Poll an ArcGIS Server for Features

Poll an external website for GeoJSON, JSON, or XML

Receive Features, GeoJSON, JSON, or XML on a REST endpoint

Receive RSS

Receive GeoJSON or JSON on a WebSocket

Receive Text from a TCP or UDP Socket

Subscribe to an external WebSocket for GeoJSON or JSON

Watch a Folder for new CSV, GeoJSON, or JSON Files

Ou

t o

f th

e B

ox

REST

.csv

WS

WS

HTTP

Es

riG

all

ery

ActiveMQ

CAP

Instagram

Exploitation Support Data

Cursor-on-Target

RabbitMQ

NMEA 0183

MQTT

Sierra Wireless (RAP)

KML

Kafka

Trimble (TAIP)

Twitter

Pa

rtn

er

Ga

llery

CompassLDE

enviroCar

GNIP

FAA (ASDI)

exactEarth AIS

Waze

Valarm

Networkfleet

OSIsoft *

*

*

*

Zonar *

Amazon IoT

Azure IoT

you can create

your own

connectors

Ingestionconfigure a input connector by pairing a transport & adapter together

GeoEvent Server

Inp

uts

Ou

tpu

ts

GeoEvent Services

Feature-JSON

Connectors Transport Adapter

GeoJSON

Receive Feature-JSON from Kafka

Receive GeoJSON on a REST endpoint

Feature-JSON

Adapters

GeoJSON

JSON

RSS

Text

XML

CAP

Cursor-on-Target

GeoMessage

Instagram

NMEA

Sierra Wireless (RAP)

Trimble (TAIP)

Twitter

VMF

Es

riG

all

ery

Waze

Feature Service

Transports

File

HTTP

HTTP+BasicAuth

HTTP+OAuth

TCP

UDP

Waze

WebSocket

Es

riG

all

ery Amazon IoT

Azure IoT

IRC

Kafka

MQTT

Kafka

HTTP

RabbitMQ

Twitter

you can create

your own

connectors

Disseminationuse an existing output connector

GeoEvent Server

Inp

uts

Ou

tpu

ts

GeoEvent Services

Ou

t o

f th

e B

ox

Add or Update a Feature

Publish Text to a UDP Socket

Send a Text Message

Send an Email

Push Text to an external TCP Socket

Push GeoJSON or JSON to an external WebSocket

Push GeoJSON or JSON to an external Website

Send an Instant Message

Send Features to a Stream Service

Write to a CSV, GeoJSON, or JSON File .csv

WS

im

HTTP

Add a Feature to a spatiotemporal big data store

Update a feature in a spatiotemporal big data store

ActiveMQ

Es

riG

all

ery

Cursor-on-Target

Hadoop

Kafka

MongoDB

MQTT

RabbitMQ

Twitter

Amazon IoT

Azure IoT

GeoEvent Server Java SDKCreate your custom GeoEvent components

Applying Real-Time Analytics

• A GeoEvent Service defines the flow of GeoEvents

- The Filtering and Processing steps to perform

- what input(s) to apply them to

- and what output(s) to send the results to

GeoEvent Services

Performing continuous analytics in ArcGISProcessors

You can create

your own

processors.

• You can perform continuous analytics on events as they are received using a processor.

GeoEvent Extension

Inp

uts

Ou

tpu

ts

GeoEvent Services

Buffer Creator

Convex Hull Creator

Difference Creator

Envelope Creator

Field Calculator

Field Enricher

Field Mapper

Geotagger

Incident Detector

Intersector

Projector

Simplifier

Symmetric Difference

Track Gap Detector

Field Reducer Union Creator

Ou

t o

f th

e B

ox

Add XYZ

Es

riG

all

ery

Bearing

Ellipse

Event Volume Control

Extent Enricher

Field Grouper

GeoNames Lookup

Range Fan

Reverse Geocoder

Service Area Creator

Symbol Lookup

Track Idle Detector

Unit Converter

Visibility

Motion Calculator Query Report

Additional Real-Time Spatial Analytics

• Spatial operators:

Spatial condition evaluation

inside outside

enter exit

intersects disjoint

touches contains

crosses equals

overlaps within

Additional Real-Time Spatial AnalyticsExample – Impending weather notifications

Additional Real-Time Spatial AnalyticsExample – Convoy separation alerting

Additional Real-Time Spatial AnalyticsGeoFence scope

• All GeoFences• Any GeoFence

• Specify a GeoFence scope: Any or All

Additional Real-Time Spatial AnalyticsExample – Territory adherence alerting

GeoEvent Server REST API documentationNew at 10.5.1

Real-Time & Big Data @ 10.5.x3

ArcGIS Enterprisewith real-time & big data capabilities

ingest visualize

analyze store

real-time & big data

millions4K e/s

4K e/s 10Ks e/s

ArcGIS

Enterprise

GeoEvent

Server

spatiotemporal

big data store

Big DataIoT

4K e/s 10Ks e/s

GeoAnalytics

Server

ArcGIS

Enterprise

GeoEvent

Server

spatiotemporal

big data store

Big Data

GeoAnalytics

Server

Writing analytic results

• GeoAnalytics Server has the ability to:

- perform analytics against data sources in the SBDS

- and write the analytic results to the SBDS

from GeoAnalytics Server

IoT

new Web GIS

layers

ArcGIS

Enterprise

GeoEvent

Server

spatiotemporal

big data store

GeoAnalytics

Server

big datafile shares

featureservice

.shp

shapefiles

.csv

text files

Hadoopfile system

Hive

Writing analytic results

• GeoAnalytics Server has the ability to:

- perform analytics against files in an external big data file share

- and write the analytic results to the spatiotemporal big data store

from GeoAnalytics Server

new Web GIS layers

Web GIS layers

Visualization4

On-the-fly aggregations

• The spatiotemporal big data store enables you to aggregate data on-the-fly

- aggregating on-the-fly accommodates for real-time data

- you can perform exploratory analysis via any combination of space, time and attribute queries

- toggle from aggregation to raw feature view

of observations

Preparing data for on-the-fly aggregations

• as data is written to a data source in the spatiotemporal big data store:

- up to four types of spatial indices are supported: geohash, square, pointy & flat hexagon/triangle

- this is in addition to a temporal index on the time field

- and an inverted index on each of the attribute fields

spatial indexing

geohash

square

trianglepointy

triangleflat

hexagonflat

hexagonpointy

Two patterns

GeoEvent Server

Your

Applications

Stream Layer

Map Services

Feature Services

Feature Layer

feature layers

Update a Feature

Add a Feature

EGDB/

SBDS

Polling

(Pull)

Stream Services

Getting Real-Time Data into Web Apps

• Feature layers pull from feature services

- Web apps poll to get periodic updates

- Must be backed by an enterprise geodatabase (EGDB) or spatiotemporal big data store (SBDS)

• Stream layers subscribe to stream services

- Web apps subscribe to immediately receive data

- Low latency and high throughput

Send Features to a Stream Service

ArcGIS Enterprise

• Ingest high velocity real-time

data into ArcGIS

• Perform continuous analytics

on events as they are received

• Store observations in a

spatiotemporal big data store

• Run batch analytics on

stored observations

• Visualize high velocity &

volume data:

- as an aggregation

- as discrete features

• Notify those who need to

know about patterns of

interest

with real-time & big data capabilities

DesktopWeb Device

visualization

live & historic

aggregates & features

map & feature servicestream service

live features

ArcGIS

Enterprise

GeoEvent

Server

spatiotemporal

big data store

storage

ingestion

analytics

GeoAnalytics

Server

analytics

Road Ahead (10.6.x)5

ArcGIS Enterprisewith real-time & big data capabilities

ArcGIS

Enterprise

GeoEvent

Server

spatiotemporal

big data store

Big DataIoT

GeoAnalytics

Server

ArcGIS Enterprisewith real-time & big data capabilities

ArcGIS

Enterprise

spatiotemporal

big data store

Big Data

GeoAnalytics

Server

IoT

RESILIENT environment

• GeoEvent 10.5

- Resiliency (high availability) & scalability is only possible if User’s “bring their own gateway”.

- Barrier to entry is HIGH & typically requires a professional services engagement for success.

bring your own

gatewayGeoEvent

Server

ArcGIS Enterprisewith real-time & big data capabilities

ArcGIS

Enterprise

spatiotemporal

big data store

Big DataIoT

GeoAnalytics

Server

GeoEvent

Server

gatewaygateway gateway

• GeoEvent 10.6

- Provides users with a resilient & scalable Real-Time GIS deployment OUT-OF-THE-BOX.

GeoEvent Server10.6 features

• RASP (Reliability, Availability, Scalability, Performance)

- Kafka as a new gateway service (dedicated process that runs Apache Kafka + Zookeeper in it’s own

life-cycle).

- Create a multi-machine GeoEvent Server site

- Uses ArcGIS Server framework for creating multi-machine site

GeoEvent Server10.6 features

GeoEvent Server10.6 features

GeoEvent Server10.6 features

GeoEvent Server10.6 features

GeoEvent Server10.6 features

• RASP (Reliability, Availability, Scalability, Performance)

- Kafka as a new gateway service (dedicated process that runs Apache Kafka + Zookeeper in it’s own

life-cycle).

- Create a multi-machine GeoEvent Server site

- Uses ArcGIS Server framework for creating multi-machine site

• 3rd party library updates for Apache Karaf, Elasticsearch & Spark

• Official Esri positioning on IoT (first time presented at UC, see session this afternoon)

• New and updated Connectors to Azure IoT, Kafka, Waze, GTFS, OAuth2 HTTP inbound

Spatiotemporal big data store10.6 enhancements

• Enhanced data retention purge rules and feature deletion

- with ability to specify a WHERE clause

• On-the-fly aggregation supports polygon, polyline & multipoint

Spatiotemporal big data storeOn-the-fly aggregation supports polygon, polyline & multipoint

Spatiotemporal big data store10.6 enhancements

• Enhanced data retention purge rules and feature deletion

- with ability to specify a WHERE clause

• On-the-fly aggregation supports polygon, polyline & multipoint

• Added space-time volume on-the-fly aggregation support

projecton-the-fly & interactive space-time aggregation

• leverage existing geohash, square & hexagon aggregation capabilities of

the spatiotemporal big data store to construct on-the-fly space-time volumes.

- enables exploratory & interactive analysis of real-time & historic observation data

ArcGIS

Enterprise

spatiotemporal

big data store

IoT

GeoEvent

Server

ArcGIS API for

JavaScript

& WebGL

client-side rendering ofspace-time volumes via WebGL

lodType=flatHexagon&lod=2&time=1474675200000,1475279999000

&timeInterval=1&timeUnit=hour

on-the-fly & interactive space-time aggregation feature service

space-time volume (based on a geohash aggregation) response

project

feature servicelodType=geohash&lod=2

geohash aggregation response

lodType=geohash&lod=2&time=1474675200000,1475279999000&timeInterval=1&timeUnit=hour&f=json

an array of time-slices of aggregated features

&returnGeometry=false &returnGeometry=false

ArcGIS EnterpriseTechnologies

ArcGIS

Enterprise

spatiotemporal

big data store

Big DataIoT

GeoAnalytics

Server

GeoEvent

Server

gatewaygateway gateway

R&D Project Trinity, Polly & Banzai6

project Polly

• as data is written to the spatiotemporal big data store:

- a spatial index for polygon aggregation is continuously updated

spatial indexing: polygon

polygonstate

polygoncounty

polygon aggregation (based on a county polygon index)polygon aggregation (based on a state polygon index)

visualization on-the-fly aggregation capabilities

Spatiotemporal,into geohash, square or hexagon

Spatial, into geohash, square or hexagon

Spatiotemporal, into polygons

Spatial, into polygons

• project Trinity is a capability that

enables existing customers and

prospects with the ability to:

- Reliably ingest, analyze & store up to

millions of sensor events per second.

- Quickly visualize, replay &

explore observations that

are billions of features.

- Perform fast analytics

on billions of observations.

an ArcGIS Enterprise bundled with massive real-time & big data capabilities

Sensors

project Trinity

DesktopWeb Device

ArcGIS Enterprise with real-time & big data

spatiotemporal

archive

real-time

batch

sources

hubs

project Trinity

on Amazon C2S

on Amazon

on Azure

Internet of your Things

bringing geospatial insights to your

DesktopWeb Device

ArcGIS Enterprise with real-time & big data

sources hubs real-time analytics spatiotemporal archive batch analytics

8 cores,28 GB ram,200 GB ssd

8 cores,28 GB ram,800 GB ssd

16 cores, 56 GB ram, 1 TB ssd

2 cores,4 GB ram,50 GB ssd

2 cores,10 GB ram,200 GB ssd

4 cores,14 GB ram,150 GB ssd

8 cores, 28 GB ram, 400 GB ssd

project Trinity

f( ) Sources + Hubs +

Real-Time Analytics f( )Spatiotemporal Archive

+ Batch Analytics

30 nodes480 cores, 1.6 TB ram, 30 TB ssd

50 nodes400 cores, 1.4 TB ram, 20 TB ssd

project Trinity

on Amazon C2S

on Amazon

on Azure

bringing geospatial insights to your

DesktopWeb Device

Internet of your Things

sources hubs real-time analytics spatiotemporal archive batch analytics

ArcGIS Enterprise with real-time & big data

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

project Trinity

1 1 11 11 1 1 1 1

1 1 11 1 1 1 1 1 1

2 2 2 2 2 2 2 2 2 2

2 2 2 2 2

3 3 3 4 4 5 6 6 6 66 67 7 6 67 7

1 1 1 1 1 12 2 2 3 3 4

ArcGIS

Enterprise

880 cores, 3TB memory, 50TB storage

project Trinity

240 cores, 1.1TB ram, 18TB storage

ArcGIS

Enterprise

monitor admin

Trinitytask manager

monitor admin

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

8 cores

28 GB ram

400 GB ssd

big data analytic pipelining & scheduling

Banzai

project Trinity

Trinity

Source ingestion

Eventhub

Real-Time analytic

Spatiotemporalarchive

Batchanalytic

Mapservice

0 to many

a lightweight

“GeoEvent Input”

distributed micro-service

0 to few

a lightweight

“GeoEvent event hub”

distributed micro-service

0 to many

a lightweight

“GeoEvent Service”

distributed micro-service

0 to few

a lightweight

“spatiotemporal big data store”

distributed micro-service

0 to many

a lightweight

“GeoAnalytics”

distributed micro-service

0 to many

a lightweight

“map & feature service”

distributed micro-service

Thank You to Our Generous Sponsor