17
Feature Geo Analytics and Big Data Processing: Hybrid Approaches for Earth Science and Real-Time Decision Making Mansour Raad, Erik Hoel, Michael Park, Adam Mollenkopf, Dawn J. Wright Environmental Systems Research Institute (aka Esri) IN12A-01 (Invited) AGU Fall Meeting, 12 December 2016

Feature Geo Analytics and Big Data Processing: Hybrid Approaches for Earth Science and Real-Time Decision Support

Embed Size (px)

Citation preview

Page 1: Feature Geo Analytics and Big Data Processing: Hybrid Approaches for Earth Science and Real-Time Decision Support

Feature Geo Analytics and Big Data Processing:

Hybrid Approaches for Earth Science and Real-Time Decision Making

Mansour Raad, Erik Hoel, Michael Park, Adam Mollenkopf, Dawn J. Wright

Environmental Systems Research Institute (aka Esri)

IN12A-01 (Invited)

AGU Fall Meeting, 12 December 2016

Page 2: Feature Geo Analytics and Big Data Processing: Hybrid Approaches for Earth Science and Real-Time Decision Support

What is Feature Geo Analytics?

A new way of processing spatiotemporal data designed for WEB-BASED big data by leveraging distributed analytics and storage

• Works with existing GIS data and tabular data

• Designed to perform both spatial and temporal analysis

• Uses familiar workflows to complete complex analyses

• “Hybridity” - integrating open-source frameworks on clusters to run analytics

Page 3: Feature Geo Analytics and Big Data Processing: Hybrid Approaches for Earth Science and Real-Time Decision Support

Feature Geo Analytics

Geoprocessing

Distributed analytics and storage

Feature Geo Analytics

Portal

Web GIS Layers

newmore extends

Page 4: Feature Geo Analytics and Big Data Processing: Hybrid Approaches for Earth Science and Real-Time Decision Support

Solve New Problems

Run analytics:

• against data too big for a single desktop machine

- Buffer 8.2 million points or thousands of polygons in a little over a minute

- billions of observations of ship movements ingested via GeoEvent

• designed to gain insight into both spatial and temporal patterns

• against massive collections in a scalable manner

• and meet time constraints

months weeks days hours minutes

Page 5: Feature Geo Analytics and Big Data Processing: Hybrid Approaches for Earth Science and Real-Time Decision Support

Geo Analytics Architectural Overview

Portal

Web GIS Layers

Un-Managed Data

New Web GIS Layers

Register large data stores, then distribute spatial analysis across cluster of machines for parallel processing

Store and/or deploy to web

Web GIS layers via Pro, Portal,

Python Notebooks, or the REST API

Managed Data

Relational Data Store

SpatiotemporalData Store

FilesFiles

Delimited Files EnterpriseShapefiles Big Data Stores

Server

Cluster

Page 6: Feature Geo Analytics and Big Data Processing: Hybrid Approaches for Earth Science and Real-Time Decision Support

Rich Collection of (Web) Analysis Tools

Summarize DataAggregate PointsSummarize NearbySummarize WithinReconstruct TracksJoin Features

Find LocationsFind Existing LocationsFind Similar Locations

Analyze PatternsCalculate DensityFind Hot SpotsCreate Space Time Cube

Use ProximityCreate Buffers

Manage DataExtract Data

* Temporally aware tools

Aggregate Points

Summarize Nearby

Summarize Within

Find Existing Locations

Find Similar Locations

Calculate Density

Find Hot Spots

Create Buffers

Extract Data

Page 7: Feature Geo Analytics and Big Data Processing: Hybrid Approaches for Earth Science and Real-Time Decision Support

Analytical Overview: Aggregating and Summarizing

• Spatial Joins

• Space-time slices

Page 8: Feature Geo Analytics and Big Data Processing: Hybrid Approaches for Earth Science and Real-Time Decision Support

• Spatiotemporal joins

Target Features Join Features Intermediate Result Final Result

Analytical Overview: Aggregating and Summarizing

Page 9: Feature Geo Analytics and Big Data Processing: Hybrid Approaches for Earth Science and Real-Time Decision Support

Temporal Relationships on Intervals

Page 10: Feature Geo Analytics and Big Data Processing: Hybrid Approaches for Earth Science and Real-Time Decision Support

• Points into Bins

Analytical Overview: Aggregating and Summarizing

Page 11: Feature Geo Analytics and Big Data Processing: Hybrid Approaches for Earth Science and Real-Time Decision Support

Aggregation – Polygons vs Cells

Aggregation By Polygons Aggregation By Cells

Page 12: Feature Geo Analytics and Big Data Processing: Hybrid Approaches for Earth Science and Real-Time Decision Support

• Reconstruct Tracks

- Summarize time-enabled points into tracks

Analytical Overview: Aggregating and Summarizing

Page 13: Feature Geo Analytics and Big Data Processing: Hybrid Approaches for Earth Science and Real-Time Decision Support

Use Case: Hurricane Tracts

• Hurricane dataset

- 120,000 points, ~100 years

- Each point has:

- ID number

- Location

- Date

- Wind speed and pressure attributes

- Problems?

- Difficult to visualize that many points

- Difficult to visualize hurricane path

Page 14: Feature Geo Analytics and Big Data Processing: Hybrid Approaches for Earth Science and Real-Time Decision Support

“Hybridity” for Distributed Computation

See also www.esri.com/software/open

Page 15: Feature Geo Analytics and Big Data Processing: Hybrid Approaches for Earth Science and Real-Time Decision Support

“Hybridity” for Distributed Computation

See also www.esri.com/software/open

Page 16: Feature Geo Analytics and Big Data Processing: Hybrid Approaches for Earth Science and Real-Time Decision Support

Real-Time GIS PerformanceArcGIS 10.4

10s of thousands of e/s

ArcGIS Spatiotemporal

Big Data Store

DesktopWeb Device

ArcGIS Server

4,000

e/s

Ingestion

GeoEvent

4,000

e/sVisualization

Live and Historic

Aggregates & Features

Enhanced Map and

Feature Service

• Ingest high-velocity real-

time data

• Observations in a Big Data

Store

• Visualize high-velocity,

high-volume data

- as an AGGREGATION,

- as discrete FEATURES,

- live & HISTORICALLY

• Visualizations CAN scale

Stream Service

Stream Layer

3,000

e/s

Live Features

Geo Analytics Performance

Spatiotemporal

Big Data Store

Page 17: Feature Geo Analytics and Big Data Processing: Hybrid Approaches for Earth Science and Real-Time Decision Support

Discussion groups at geonet.esri.com

Step 1. Click orange “Join in” button to create your

account.

Step 2. Join the Big Data or Sciences groups

Step 3. Contribute to AGU conversations!

Mansour Raad, Esri Big Data Team

[email protected]

thunderheadxpler.blogspot.com

github.com/mraad

@mraad

For Questions/Discussion