ArcGIS GeoEvent Server: Best Practices2012 R2 Standard & Datacenter (Sep 2016) 2012 Standard & Datacenter (Sep 2016) 2008 R2 Standard, Enterprise & Datacenter (sp1) 2008 Standard,

GeoEvent Server:Best Practices

Frank RobertsDirector of IT Services, Innovate

[email protected]

Adam MollenkopfReal-Time & Big Data GIS Capability Lead, Esri

[email protected]

@amollenkopf

Suzanne FossReal-Time & Big Data Product Engineer, Esri

[email protected]

@sfoss_esri

mailto:[email protected]

1

2

3

4

5

6

Agenda

Innovate: one-stop real-time air monitoring

ingestion & analytics

dissemination & visualization

storage

resilience & scalability

deploying at a massive scale

Innovate1 one-stop real-time air monitoring

INNOVATE!, Inc.AMCO Chemical Superfund Site

One-Stop Real-Time Air Monitoring

February 14, 2017

1

2

3

4

AGENDA

About Us

Big Data, Dig Deep

Beware of the Quirks

From the old, come the new

Innovate is a geographically diverse 8(a) woman-minority owned company, with a primary officelocation at the Ronald Reagan Building. Two primaryservice areas have garnering multiple awards forinnovation and partnership:

• Geospatial Solutions

• Information Technology

Awards include:

2014 EPA Mason Hewitt Award for Excellence,EnviroAtlas Application

2012 EPA Outstanding 8(a) Small Business of the Year

2010 EPA/CIO Award for Innovation, GeoDataGateway

ABOUT US

1

Ground SensorsMC2

OTIE Scribe

MC2 Scribe

Viper

Field/ Analytical Collection

Data Logging

Data Logging

Publish

Publish

Publish

EDMS

MC2 Database

AMCOScribe

Replication

OTIE

AMCO Web

Interface

Web Server

EPA RPM

Public

EPA Start

Inte

rnet

Bro

wse

rs

Subscription

Merged AMCO Scribe

R9 Scribe Replication

at R9ArcServer

AMCO Service

R9 Scribe/Viper

Replication at ER Cloud

EPA GIS Center Subscription

(On ER Cloud) (On Site at R9)

Air QualityViper OData Feed

Scribe.Net

What• Monitor the site• Provide status updates

(public and internal)

Why• Track cleanup progress• Ensure public safety

How• Harness the beast by any

means necessary(But use existing EPA tools)(But do it in a map)(That we can open anywhere)

(Include a unicorn)

Big Data, No Problem

Deep in the Data…Identify the required objects in the JSON structure

Deeper in the Data…Data structure dictates Input requirements

• Select directly into CurrentReadings object

• Create a new GeoEvent Definition

• Build Geometry from Fields (Lat/Long)

GeoEvent Definitions

Use your Input Connector to create a GeoEvent Definition, then create a copy and tweak as necessary for the output

THEN create tables/feature classes in the database according to the definition

1

2

3

Assign tags to the output definition

Feature classes are flat – don’t give them hierarchical data

GeoEvent Services

Putting points on the mapMove from GeoEvents to a fully integrated Story Map with live data

Logs don’t always tell the whole story!Where’s the data?

In’s and Out’s go up, but no data in the database?

Use a text or json output to test your processorsand filters…

- See what you expect?

- Are datatypes correct?

Time Zone Time Warp

Next… Find the most recent reading from each sensor and publish as a feature class…

…but beware of time zones.

Sensor time ≠ Database Server time ≠ User time

• Set Windows tasks to delete old data (not GeoEvents)

• Convert feature class to UTC so AGOL popup displays is correct

Value Added

Use the ArcGIS Online tools we have... Build only the tools we don’t…

Value Added

https://epa.maps.arcgis.com/apps/MapSeries/index.html?appid=2e0a0e3d5cc041feb0c19d9f04327bcb


Quick Links

AMCO OSC:

https://www.epaosc.org/site/site_profile.aspx?site_id=5215

AMCO Story Map:


(requires a GeoPlatform login)

AMCO Graphing Site:

r9.ercloud.org/AMCO

VIPER:

https://viper.ert.org/DeploymentManager/About.aspx

Scribe:


More information about Esri GeoEvents:

http://www.esri.com/software/arcgis/arcgisserver/extensions/geoevent-extension



r9.ercloud.org/AMCO

https://viper.ert.org/DeploymentManager/About.aspx


http://www.esri.com/software/arcgis/arcgisserver/extensions/geoevent-extension

1

2

3

4

5

6

Agenda

Innovate: one-stop real-time air monitoring

ingestion & analytics

dissemination & visualization

storage

resilience & scalability

deploying at massive scale

ingestion & analytics2

you can create

your own

connectors

Ingestionuse an existing input connector

GeoEvent Server

Inp

uts

Ou

tpu

ts

GeoEvent Services

Poll an ArcGIS Server for Features

Poll an external website for GeoJSON, JSON, or XML

Receive Features, GeoJSON, JSON, or XML on a REST endpoint

Receive RSS

Receive GeoJSON or JSON on a WebSocket

Receive Text from a TCP or UDP Socket

Subscribe to an external WebSocket for GeoJSON or JSON

Watch a Folder for new CSV, GeoJSON, or JSON Files

Ou

t o

f th

e B

ox

REST

.csv

WS

WS

HTTP

Esri

Gallery

ActiveMQ

CAP

Instagram

Exploitation Support Data

Cursor-on-Target

RabbitMQ

NMEA 0183

MQTT

Sierra Wireless (RAP)

KML

Kafka

Trimble (TAIP)

Twitter

Pa

rtn

er

Ga

lle

ry

CompassLDE

enviroCar

GNIP

FAA (ASDI)

exactEarth AIS

Waze

Valarm

Networkfleet

OSIsoft *

*

*

*

Zonar *

you can create

your own

connectors

Ingestionconfigure a new input connector by pairing a transport & adapter together

GeoEvent Server

Inp

uts

Ou

tpu

ts

GeoEvent Services

Feature-JSON

Connectors Transport Adapter

GeoJSON

Receive Feature-JSON from Kafka

Receive GeoJSON on a REST endpoint

Feature-JSON

Adapters

GeoJSON

JSON

RSS

Text

XML

CAP

Cursor-on-Target

GeoMessage

Instagram

NMEA

Sierra Wireless (RAP)

Trimble (TAIP)

Twitter

VMF

Esri

Gallery

Waze

Feature Service

Transports

File

HTTP

HTTP+BasicAuth

HTTP+OAuth

TCP

UDP

Waze

WebSocket

Esri

Gallery ActiveMQ

IRC

Kafka

MQTT

RabbitMQ

Twitter

Kafka

HTTP

Ingestionclone an existing connector & tailor it’s properties to make a more specific connector

Ingestionclone an existing connector & tailor it’s properties to make a more specific connector

Ingestionextend GeoEvent by downloading additional transports & adapters

https://github.com/esri/solutions-geoevent-java

http://links.esri.com/geoevent-gallery

AnalyticsGeoEvent Services best practices

input1 output1buffer

• Which is better?

- have fewer but more complex GeoEvent Services

- or have more but simpler GeoEvent Services

geofences

filter(spatial)

input1 output2

geofences

filter(spatial)

geofences

geotagger

input2 output3

geofences

filter(spatial)

motioncalculator

geometry inside Zones/.*

geometry inside Zones/.* geometry inside Zones/.*


AnalyticsGeoEvent Services best practices

• Which is better?

- have fewer but more complex GeoEvent Services

- or have more but simpler GeoEvent Services

input1 output1buffer

geofences

filter(spatial)

input2 output2

geofences

geotagger

output3motion

calculator



dissemination & visualization3

you can create

your own

connectors

Disseminationuse an existing output connector

GeoEvent Server

Inp

uts

Ou

tpu

ts

GeoEvent Services

Ou

t o

f th

e B

ox

Add or Update a Feature

Publish Text to a UDP Socket

Send a Text Message

Send an Email

Push Text to an external TCP Socket

Push GeoJSON or JSON to an external WebSocket

Push GeoJSON or JSON to an external Website

Send an Instant Message

Send Features to a Stream Service

Write to a CSV, GeoJSON, or JSON File .csv

WS

im

HTTP

ActiveMQ

Esri

Gallery

Cursor-on-Target

Hadoop

Kafka

MongoDB

MQTT

RabbitMQ

Twitter

Add a Feature to a spatiotemporal big data store

Update a feature in a spatiotemporal big data store

you can create

your own

connectors

Disseminationconfigure a new output connector by pairing an adapter & transport together

GeoEvent Server

Inp

uts

Ou

tpu

ts

GeoEvent Services

Message Formatter SMTP

Connectors TransportAdapter

Feature-JSON Kafka

Send an Email

Publish Feature-JSON to Kafka

Big Data Store

Transports

Feature Service

File

HTTP

HTTP+BasicAuth

HTTP+OAuth

SMTP

Stream Service

TCP

UDP

WebSocket

XMPP

Cache

Adapters

Feature-JSON

GeoJSON

JSON

Message Formatter

RSS

Text

WebSocket

ActiveMQ

Hadoop

IRC

Kafka

MongoDB

MQTT

Esri

Gallery

RabbitMQ

TCP-Squirt

Twitter

Cursor-on-Target

Esri

Gallery

SMS

you can create

your own

connectors

GeoEvent Server

Inp

uts

Ou

tpu

ts

GeoEvent Services

Ou

t o

f th

e B

ox


Publish Text to a UDP Socket

Send a Text Message

Send an Email

Push Text to an external TCP Socket

Push GeoJSON or JSON to an external WebSocket

Push GeoJSON or JSON to an external Website

Send an Instant Message

Send Features to a Stream Service

Write to a CSV, GeoJSON, or JSON File .csv

WS

im

HTTP

ActiveMQ

Esri

Gallery

Cursor-on-Target

Hadoop

Kafka

MongoDB

MQTT

RabbitMQ

Twitter

Add a Feature to a spatiotemporal big data store

Update a Feature in a spatiotemporal big data store

Visualizationvisualizing real-time observation results using Esri clients

Visualizationchoosing a service type: stream service, feature service, map service

Stream Layer

Map Layer

Feature Layer

• Stream layers in apps subscribe to stream services to immediately visualize observations

- does not require storage, low latency, no playback

• Map & Features layers in apps periodically poll to visualize most current observations

- backed by an enterprise geodatabase (EGDB) or a spatiotemporal big data store (BDS)

- history can be retrieved & queried for playback

desktop

device

web

ArcGIS

Enterprise

Stream Service

Send Features to a Stream Service subscribe (push)

polling (pull)

Map Service

Feature Service

GeoAnalytics

Server

Add a Feature to a BDS

Update a Feature in a BDS

spatiotemporal

big data store


EGDBGeoEvent

Server

Visualizationworkflow for creating a real-time service using stream service, feature service, map service

• Configure an input to receive real-time data & define a geoevent definition accordingly

• Create an output and publish a Stream / Feature / Map Service using the geoevent definition

• Author and publish a geoevent service

• Visualize your Stream / Feature / Map Service

ArcGIS Enterprise

spatiotemporal

big data storeGeoEvent

Server

input

geoevent

definition

geoevent service

Stream Service

Map Service

Feature Service

output

storage4

resource minimum RECOMMENDED notes

64-bit

operating system

(Windows)

Windows Server:

2012 R2 Standard & Datacenter (Sep 2016)

2012 Standard & Datacenter (Sep 2016)

2008 R2 Standard, Enterprise & Datacenter (sp1)

2008 Standard, Enterprise & Datacenter (sp2)

Windows Server:

2012 R2 Standard & Datacenter (Sep 2016)

‘_’ characters are

not allowed in

machine names

64-bit

operating system

(Linux)

Red Hat Enterprise Linux Server 7 (update 8)

Red Hat Enterprise Linux Server 6 (update 2)

CentOS Linux 7 (7.2)

CentOS Linux 6 (6.8)

Scientific Linux 7 (7.2)

Scientific Linux 6 (6.8)

Ubuntu Server LTS (16.04.1)

SUSE Linux Enterprise Server 12 (sp1)

SUSE Linux Enterprise Server 11 (sp4)

Oracle Linux 7 (update 2)

Oracle Linux 6 (update 8)

Use your Linux flavor of choice with the

latest update version Esri has certified.

http://server.arcgis.com/en/data-store/latest/install/windows/arcgis-data-store-system-requirements.htm

http://server.arcgis.com/en/data-store/latest/install/linux/arcgis-data-store-system-requirements.htm

spatiotemporal big data storeoperating system requirements

resource requirements: processors




processors 8 cores 16 cores

More processors results in

faster writes (indexing time),

higher write throughput &

faster query response time.

spatiotemporal big data store

resource requirements: processors, memory









storage 400 GB 1 TB or moreStorage need is dictated by data velocity & size.

SSD results in much faster writes & queries.


resource requirements: processors, memory, storage











memory 16 GB 32 GBMore memory results in

higher likelihood of query cache hits.

32 GB is the maximum due to JVM limitations.


resource requirements: processors, memory, storage & instances









memory 16 GB 32 GBMore memory results in

higher likelihood of query cache hits.

32 GB is the maximum due to JVM limitations

# of

ISOLATED

instances

1 3 or more

With one instance there is risk of data loss.

With three instances replicas are written

increasing data reliability.

The # of instances will vary based on the velocity of

incoming data as well as the data retention policy

configured on each data source.




spatiotemporal

big data store

GeoEvent

Server

shards & replication factor

GeoAnalytics

Server

node 1

node 2

node 3node 4

node 5

T1

T3

T2

T2

r = 1

T1

T3


spatiotemporal

big data store

auto-rebalancing of data upon node membership changes, + or -, in the big data store

GeoEvent

Server

GeoAnalytics

Server

node 2

node 3node 4

node 5

T1

T3

T2

T2

r = 1

x

T1

T3

T1

T1T3


purge based on

data retention

spatiotemporal

big data store

GeoEvent

Server

data retention policies, configured per data source

GeoAnalytics

Server

node 1

node 2

node 3node 4

node 5

r = 1


purge based on

data retention

spatiotemporal

big data store

GeoEvent

Server

rolling index option, set appropriately to the velocity of your observation data

GeoAnalytics

Server

node 1

node 2

node 3node 4

node 5

r = 1

indices indices

indices indices

indices


purge based on

data retention

spatiotemporal

big data store

GeoEvent

Server

automatic data backups using periodic snapshots, including ability to restore from a snapshot

GeoAnalytics

Server

node 1

node 2

node 3node 4

node 5

r = 1

snapshot-2016-05-17-11-0-0.snapshot

snapshot-2016-05-17-12-0-0.snapshot

…


choosing an Object Id optionspatiotemporal big data store

choosing an Object Id optionspatiotemporal big data store

choosing an Object Id option

Max Value # of IDs ArcGIS Clients

Int32 2,147,483,647 2.1 billion Pro, Desktop, Ops Dashboard, …

events per day Int32

1,000 e/s 86,400,000 25 days

10,000 e/s 864,000,000 2.5 days

100,000 e/s 8,640,000,000 6 hours

1,000,000 e/s 86,400,000,000 36 minutes

10,000,000 e/s 864,000,000,000 4 minutes





Int64 (signed) 9,223,372,036,854,775,807 9.2 quintillion JavaScript, custom apps

events per day Int32

1,000 e/s 86,400,000 25 days

10,000 e/s 864,000,000 2.5 days

100,000 e/s 8,640,000,000 6 hours

1,000,000 e/s 86,400,000,000 36 minutes

10,000,000 e/s 864,000,000,000 4 minutes






events per day Int32 Int64 (signed)

1,000 e/s 86,400,000 25 days 292,472,000 years

10,000 e/s 864,000,000 2.5 days 29,247,200 years

100,000 e/s 8,640,000,000 6 hours 2,924,720 years

1,000,000 e/s 86,400,000,000 36 minutes 292,472 years

10,000,000 e/s 864,000,000,000 4 minutes 29,248 years






UniqueStringID n/a unlimited JavaScript, custom apps

events per day Int32 Int64 (signed)

1,000 e/s 86,400,000 25 days 292,472,000 years

10,000 e/s 864,000,000 2.5 days 29,247,200 years

100,000 e/s 8,640,000,000 6 hours 2,924,720 years

1,000,000 e/s 86,400,000,000 36 minutes 292,472 years

10,000,000 e/s 864,000,000,000 4 minutes 29,248 years


resilience & scalability5

best practices

000

GeoEvent resilience & scalability

• GeoEvent clustering was introduced at 10.3

• However, observed to be brittle and subject to failure:

- Must have at least 3 nodes in a cluster

- when a node goes down, it must be brought back up ASAP.

- a 2-node cluster can cause a split-brain condition in cluster.

- Must be physical hardware

- many customers only have virtualized environments.

- Network must be stable

- network instabilities can result in intermittent membership.

• Given this, clustering GeoEvent

- should be avoided and is discouraged

• Instead, if resiliency or addt’l scalability is required:

- isolated deployments should be employed

- this is true at 10.3, 10.4 and 10.5

isolated deployment

• Each GeoEvent instance exists in it’s own site and shares a common configuration manually.

• A message broker must be stood up in front to provide a common event source.

• GeoEvent instances share a common message broker and run active/active competing to

consume events.


isolated deployment

planes

trains

1 1

1 1


isolated deployment

planes

trains

2

2 2

2


isolated deployment

planes

trains

1 1

1 1


isolated deployment

• Known constraints:

- does not support failover of state: enter, exit, track gap detection, idle detector

- project team/IT staff needed to setup/configure message broker and get all sensor events into it

- input to GeoEvent is the message broker only, lose flexibility of all the input connectors available.


isolated deployment

• An isolated deployment of GeoEvent instances leads to challenges with Stream Services:

- Client A & B see event 1, while client C & D do not

- Client C & D see event 2, while client A & B do not

1

1 1

1

1

2

2

2

2

2

Stream Service resilience & scalability

isolated deployment

• Using Kafka the GeoEvent instances can be configured to use separate consumer groups:

- With this configuration, all clients see all events

- note: can also be accomplished with RabbitMQ fan-outs or ActiveMQ topics

1

1 1

1

1

1 1

1

1


isolated deployment




2

2

2

2

2

2 2

2

2

1

1

1

1


isolated deployment

• A reverse proxy can be configured in between the clients and the stream services so that clients

don’t have direct knowledge of the servers they are connecting to.

- NGiNX is one such reverse proxy.

1

1 1

1 1

1

1

1

1


isolated deployment




2

2

2

2 2

2

2

1

1

1

1


deploying at a massive scale6

ArcGIS Enterprisewith real-time & big data capabilities

ArcGIS

Enterprise

Big DataIoT

GeoEvent

Server

spatiotemporal

big data store

GeoAnalytics

Server

ArcGIS

Enterprise

with real-time & big data capabilities

• A new class of customer is demanding MASSIVE real-time & big data analytic capabilities.

ArcGIS Enterprise

big data analytics & storageingestion & real-time analytics

Big DataIoT

Real-Time & Big Data GIS

Tue, 5:15pm-6:15pm, Room 151Aat a Massive Scale

SummaryGeoEvent Server: best practices

• When applying Real-Time GIS there are many best practices for

- ingestion & analytics

- dissemination & visualization

- storage

- resilience & scalability

• To learn more:

- See the tutorial:

‘ArcGIS GeoEvent Server: Resiliency’

coming soon

Real-Time & Big Data GIS

• Applying Real-Time Analytics Mon, 1:45pm-2:45pm, Room 147B

Tue, 11:00am-12:00pm, Room 151B

• Leveraging the Spatiotemporal Big Data Store Mon, 3:00pm-4:00pm, Room 147B

Tue, 1:30pm-2:30pm, Room 151B

• ArcGIS GeoAnalytics: Intro to Big Data Feature Analytics Mon, 4:15pm-5:15pm, Room 147B

Tue, 2:45pm-3:45pm, Room 147A

• Leveraging Stream Services Tue, 2:45pm-3:45pm, Room 151A

• ArcGIS GeoEvent Server: Best Practices Tue, 4:00pm-5:00pm, Room 151A

• Real-Time & Big Data GIS at a Massive Scale Tue, 5:15pm-6:15pm, Room 151A

other sessions

Please Take Our Survey on the Esri Events App!

Select the session

you attended

Scroll down to find

the survey

Complete Answers

and Select “Submit”

Download the Esri Events

app and find your event

Print Your Certificate of Attendance

Print stations located in the 140 Concourse

Monday

12:30 PM – 6:30 PM

GIS Solutions Expo,

Hall B

5:15 PM – 6:30 PM

Expo Social,

Hall B

Tuesday

10:45 AM – 5:15 PM

GIS Solutions Expo,

Hall B

6:30 PM – 9:30 PM

Networking Reception,

Smithsonian National Air

and Space Museum

Questions / Feedback?

http://links.esri.com/geoevent

http://links.esri.com/geoevent-forum

To learn more:

Frank RobertsDirector of IT Services, Innovate

[email protected]

Adam MollenkopfReal-Time & Big Data GIS Capability Lead, Esri

[email protected]

@amollenkopf

Suzanne FossReal-Time & Big Data Product Engineer, Esri

[email protected]

@sfoss_esri

mailto:[email protected]

Documents

ArcGIS GeoEvent Server: Best Practices2012 R2 Standard & Datacenter (Sep 2016) 2012 Standard & Datacenter (Sep 2016) 2008 R2 Standard, Enterprise & Datacenter (sp1) 2008 Standard,