Upload
others
View
17
Download
0
Embed Size (px)
Citation preview
Latest Developments in Oceanographic Applications of GIS
including !
Dawn Wright Esri Chief Scientist
Affiliate Professor, Oregon State U.
Near-Real-Time Interactive Map Exemplars & Scientific Empowerment Through Storytelling
Schmidt Ocean Institute Research Planning Workshop, August 26, 2015
Analytics
Dashboards
Cool Maps Storytelling Apps
Analytics!.
Apps
Access
Desktop Web Device
Services
GeoEvent Extension
GeoAnalytics Extension
A GeoAnalytics
A
ArcGIS Server
Ingestion
Visualization
A Analytics
ArcGIS Spatiotemporal Big Data Store
Services
ArcGIS Spatiotemporal
Storage
High velocity and volume spatiotemporal data
Stream Layer
Stream Service
Web & Real-Time GIS
•! Integration, analysis of high-velocity & -volume data
•! Updates, alerts sent out
Types of (Geo)Analytics
Aggregate by Polygons or Cells
Thanks to Adam Mollenkopf, Esri Real-Time GIS Capability Team Lead and Ricardo Trujillo, GeoEvent Developer, [email protected], [email protected] links.esri.com/geoevent-wiki We aim to register large data stores / data sets with ArcGIS Server, then distribute analysis across a cluster of machines for parallel processing Performance example: buffer 8.2 million points or thousands of polygons in a little over a minute, Coming: ~250,000 writes to disk per second across 5 nodes Many frameworks/technologies exist for distributing computation
E.g., Hadoop, MapReduce, Spark Spark: processes distributed data in memory; Supports MapReduce programming model
Includes additional framework level distributed algorithms ArcGIS Server integrates these technologies on a cluster to solve analytic problems
5
Parallelized batch analy0cs on tabular, vector, raster, and imagery datasets (big and standard data) Performance example: buffer 8.2 million points or thousands of polygons in a little over a minute, NetCDF output is supported on … Aggregate Points – Point in polygon aggregation (spatial join) Aggregate by Cell – simple aggregation Summarize Nearby – spatial join where left dataset and a distance are used to inflate the bounding box. A distance calculation is then employed (no drivetime/distance). Summarize Within – a spatial join that is a generalization of Aggregate Points Find Existing Locations – an attribute query followed by/with a spatial query (?); the attribute portion is SparkSQL, the spatial relationship is: intersections, within distance, contains, and within. Find Similar Locations – non-spatial similarity search (SparkSQL). Calculate Density – cell-based spatial aggregation by attribute with a weighted neighborhood/distance-based summarization. Find Hot Spots – very similar to Calculate Density but with the Getis-Ord Gi* statistic. This can be cell-based (point observations) or polygon-based (polygon source data). Create Buffers – simple spatial buffer generation across the dataset (the Java geometry library supports Euclidean and Geodetic buffer generation). This is done. Extract Data – a simple region query with a specified output format. Field (RASTER) Calculator – apply a Scala or SQL expression to each feature, one expression per field.
6
http://blogs.esri.com/esri/apl/2015/03/11/dimension-explorer/
Dimension Explorer Add-In (Desktop)
Simple Integration
Out of the Box
Interoperability / Crosswalking
r-arcgis.github.io
Interoperability / Crosswalking
Apps
Access
Desktop Web Device
Services
GeoEvent Extension
GeoAnalytics Extension
A GeoAnalytics
A
ArcGIS Server
Ingestion
Visualization
A Analytics
ArcGIS Spatiotemporal Big Data Store
Services
ArcGIS Spatiotemporal
Storage
High velocity and volume spatiotemporal data
Stream Layer
Stream Service
Web & Real-Time GIS
Dashboards!
Shin Kobara, Texas A&M
Wave Glider Sensor Dashboard: bit.ly/gcooswaveglider
From Kobara et al., 2015, Ocean Solutions, Earth Solutions
Shin Kobara, Texas A&M
Esri Global Fishing Watch aimed at Illegal, Unreported, Unregulated Fishing
Active Vessels Dashboard Prototype
Cool Maps!.
coolmaps.esri.com/BigData/ShippingTracks
coolmaps.esri.com/BigData/ShippingTracks
coolmaps.esri.com/BigData/ShippingGlobe
coolmaps.esri.com/BigData/ShippingGlobe
coolmaps.esri.com/BigData/ShippingGlobe
arcg.is/1Pg7Jlg
Once upon a time!
“Stories are data with a soul, !persuasive and memorable.” – Nancy Baron, Director of Science Outreach, COMPASS
By Charles Saxon | Published March 21, 1983 Licensed personally to Dawn Wright by The New Yorker Cartoon Bank, TCB-86966, Invoice L12480
“Story Maps” Communicating with !
esriurl.com/ocnstories
esriurl.com/ocnstories esriurl.com/ocnstories
Sensor dashboard as a story map
esriurl.com/ocnstories
esriurl.com/ocnstories
storymaps.arcgis.com
esriurl.com/ocnstories
Natural Resources Defense Council, esriurl.com/ocnstories
Natural Resources Defense Council, esriurl.com/ocnstories
Natural Resources Defense Council, esriurl.com/ocnstories
esriurl.com/scicomm, esriurl.com/ocnstories, esriurl.com/ocnres
Final Chapter
Consider implications for education, training,
certification?
Consider potential of stories and scientific storytelling via maps, photos, videos, more
Consider implications for ocean science
contributions to society
Dawn Wright [email protected]
@deepseadawn
Extra Slides
Story map made into a conference poster with interactive version on table for discussion &questions, e.g., esriurl.com/agustories
esriurl.com/analyticalstories
How do I get real-time data into a GIS? ArcGIS example: Use an Input Connector
You can create your own
connectors.
GeoEvent Extension
Inpu
ts
Out
puts
GeoEvent Services
Poll an ArcGIS Server for Features
Poll an external website for GeoJSON, JSON, or XML
Features, GeoJSON, JSON, or XML on a REST endpoint
Receive RSS
Receive GeoJSON or JSON on a WebSocket
Receive Text from a TCP or UDP Socket
Subscribe to external WebSocket for GeoJSON or JSON
Watch a Folder for new CSV, GeoJSON, or JSON Files
Esri
Out
of t
he B
ox
REST
Watch a Folder for new CSV, .csv
Poll an ArcGIS Server for Features
WS
WS
HTTP
Receive RSS
Receive Text from a TCP or UDP Socket Receive Text from a TCP or UDP Socket
E
sri G
alle
ry
ActiveMQ
CAP
Gal
lery
Instagram Gal
lery
G
alle
ry
Flume
Gal
lery
G
alle
ry
Cursor-on-Target
RabbitMQ
NMEA 0183
MQTT
Sierra Wireless (RAP)
KML
Kafka
Cursor-on-Target
*
Trimble (TAIP)
Kafka
Twitter Twitter
MQTT P
artn
er G
alle
ry
CompassLDE
enviroCar
GNIP
FAA (ASDI)
exactEarth AIS
Zonar
Valarm
Networkfleet
OSIsoft
GNIP
FAA
* * *
*
*
*
Flume *
Applying Real-Time Analytics ArcGIS example: Use a GeoEvent Processor
You can create your own
processors.
GeoEvent Extension
Inpu
ts
Out
puts
GeoEvent Services
Aggregator
Buffer Creator
Convex Hull Creator
Difference Creator
Envelope Creator
Field Calculator
Field Enricher
Geotagger
Incident Detector
Intersector
Projector
Simplifier
Sliding Time Window
Symmetric Difference
Field Mapper Track Gap Detector Esri
Out
of t
he B
ox Add XYZ
Esri
Gal
lery
Bearing
Ellipse
Event Volume Control
Extent Enricher
Field Grouper
GeoNames Lookup
Range Fan
Reverse Geocoder
Service Area Creator
Symbol Lookup
Track Idle Detector
Unit Converter
Visibility
Motion Calculator Query Report Field Reducer Union Creator
*
*
How do I update and alert those who need it where they need it? Disseminate notifications, alerts, and updates using an Output Connector
You can create your own
connectors.
GeoEvent Extension
Inpu
ts
Out
puts
GeoEvent Services
Esr
i Out
of t
he B
ox
Add or Update a feature
Publish Text to a UDP Socket
Send a Text Message
Send an Email
Add or Update a feature
Push Text to an external TCP Socket
Push GeoJSON or JSON to an external WebSocket
Push GeoJSON or JSON to an external Website
Send an Email
Send a Text Message
Send an Instant Message
Send Features to a Stream Service
Push Text to an external TCP Socket Push Text to an external TCP Socket
Publish Text to a UDP Socket Publish Text to a UDP Socket
Write to a CSV, GeoJSON, or JSON File Write to a CSV, .csv
WS
im
HTTP
Write to a Spatiotemporal Data Store Write to a Spatiotemporal Data Store
ActiveMQ
Esri
Gal
lery
Cursor-on-Target
Flume
Hadoop (HDFS)
Kafka
MongoDB
*
MQTT
Cursor-on-Target
RabbitMQ
MongoDB
Kafka
MQTT
*
Flume
*
Two patterns Getting Real-Time Data into Web Applications
•! Feature layers pull from feature services -! Web apps poll to get periodic updates -! Must be backed by an enterprise geodatabase (EGDB)
•! Stream layers subscribe to stream services -! Web apps subscribe to immediately receive data -! Low latency and high throughput
GeoEvent Extension
ArcGIS Server
Your
Applications
Stream Layer
Map Services Feature Services
Feature Layer
feature layers
Update a Feature Add a Feature
EGDB
Polling (Pull)
Stream Services
Send Features to a Stream Service