Geospatial Temporal Open Standards for Big Data from Space (BiDS2014)

®

OGC

Big Data and Open Geospatial-Temporal

Information Standards

George Percivall OGC Chief Engineer

2014 Conference on Big Data from Space (BiDS '14)

Jointly organised by ESA, SatCen, JRC 12-14 November 2014, ESRIN, Frascati, Italy

© 2014 Open Geospatial Consortium

OGC ®

OGC Services Architecture

•  Introduction to OGC Standards • …Service Interfaces and Encodings standards • …Grouped into semantically similar sets

OGC ®


WMS

WMTS WFS

Simple Features Access

Access Services

Geospatial Feature Data

Geospatial Browse/Maps

Geospatial Coverage Data

WCS

OGC ®


WMS

WMTS WFS


Access Services




WCS

Sensors

Puck

SOS SPS O&M SensorML

Sensor Web Enablement

Discover Task Access

OGC ®


Other Data

Processing Services OpenMI WPS TJS WCPS

WMS

WMTS WFS


Access Services




WCS

Sensors

Puck




OGC ®


Other Data


Geospatially Enabled Metadata

Discovery Services

CSW OpenSearch Geo ebRIM

WMS

WMTS WFS


Access Services




WCS

Sensors

Puck




OGC ®


Other Data



Discovery Services


WMS

WMTS WFS


Access Services




WCS

Other Services Workflow, Alerts

Sensors

Puck




OGC ®

SLD

SE

Data Models and Encodings WMC

FE

GML

GeoXACML

KML

CityGML OpenGeoSMS

IndoorGML GeoSparql

WaterML GeoPackage

NetCDF GMLJP2


Other Data



Discovery Services


WMS

WMTS WFS


Access Services




WCS


Sensors

Puck




OGC ®

Visualization / Decision Tools and Applications GeoAPI

OpenLS

SLD

SE

Data Models and Encodings WMC

FE

GML

GeoXACML

KML

CityGML OpenGeoSMS

IndoorGML GeoSparql

WaterML GeoPackage

NetCDF GMLJP2


Other Data



Discovery Services


WMS

WMTS WFS


Access Services




WCS


Sensors

Puck




OGC ®

ESA HMA and OGC

•  Foreword by Mark Reichardt, OGC CEO and President

–  Comprehensive guide to the use of international standards to enable a European EO system of systems.

–  “Indispensable resource for organisations with EO assets that wish to connect to and build a local to global system of systems”

•  OGC Working Groups of major importance to HMA

–  Service Architecture –  Sensor Web Enablement –  Catalogue –  Metadata –  Coverages –  Compliance

© 2014 Open Geospatial Consortium 10

Thomas Usländer (Fraunhofer IOSB) Yves Coene (Spacebel s.a.) Pier Giorgio Marchetti (ESA)

OGC ®

ESA HMA Protocols under Configuration Control http://wiki.services.eoportal.org/tiki-index.php?page=HMA+Configuration+Management+Table


Service Standard Status Collection Metadata OGC 11-035r1, EO Collection and Service Discovery using the ebRIM Application Profile of CSW 2.0, version

1.0, 26/03/2013. OGC Best Practice

Collection& Service Discovery

OGC 13-084, I15 (ISO19115 Metadata) Extension Package of CSW ebRIM profile, version 1.0, 28/04/2014 OGC Implementation Specification

OGC 10-032r8, OpenSearch Geo and Time Extensions, version 1.0.0, 14/04/2014. OGC Implementation Standard OGC 13-026r5, OpenSearch Extension for Earth Observation, DRAFT 1.0.0, 29/10/2014 OGC SWG Internal document Earth Observation Product Metadata

OGC 06-080r4, GML Application Schema for EO Products, Version 1.0.0, 25/02/2010 OGC Profile

OGC 10-157r3, Earth Observation Metadata Profile of Observations and Measurements,Version 1.0, 12/06/2012 OGC Implementation Standard Product Discovery OGC 06-131r6, EO Products Extension Package for ebRIM Profile of CSW 2.0, Version 1.0.0, 10/02/2010 OGC Implementation Standard OGC 10-032r8, OpenSearch Geo and Time Extensions, version 1.0.0, 14/04/2014. OGC Implementation Standard OGC 13-026r5, OpenSearch Extension for Earth Observation, DRAFT 1.0.0, 29/10/2014 OGC SWG Internal document Order OGC 06-141r6. Ordering Services Framework for Earth Observation Products Interface Standard, Version 1.0.0,

09/01/2012 OGC Implementation Specification

OGC 13-042, RESTful Encoding of Ordering Services Framework for Earth Observation Products, version 1.0, 28/04/2014.

OGC Best Practice

Feasibility Analysis OGC 10-135, Sensor Planning Service Interface Standard 2.0 Earth Observation Satellite Tasking Extension, version 2.0, 28/03/2011

OGC Implementation Standard

OGC 13-039, OpenSearch Extension for Earth Observation Satellite Tasking, OGC Best Practice OGC 14-012r1, RESTful encoding of Sensor Planning Service for Earth Observation Satellite Tasking, version

1.0.0, 17/07/2014. OGC Best Practice

Web Map Service (WMS)

OGC 07-063r1 OpenGIS WMS EO Application Profile Version 0.3.3, 05/11/2009 OGC Best Practice

Web Coverage Service (WCS)

OGC 09-110r4 WCS 2.0 Interface Standard - Core, Version 1.0.0, 10/07/2012. OGC 09-146r2 GML 3.2.1 Application Schema - Coverages (1.0.1) OGC 09-147r3 WCS 2.0 Interface Standard - KVP Protocol Binding Extension (1.0.1) OGC 09-149r1 WCS 2.0 Interface Standard - XML/SOAP Protocol Binding Extension (1.0) OGC 09-148r1 WCS 2.0 Interface Standard - XML/POST Protocol Binding Extension (1.0) OGC 09-153 WCS 2.0 Overview: Core and Extensions (Best Practice) OGC 10-164, OGC GML 3.2.1 AS – Coverages – CF NetCDF Coverage Encoding Extension, version 2.0.0 OGC 12-108, OGC GML 3.2.1 AS – Coverages – GML JPEG2000 Coverage Encoding Extension, version 0.0.4.

OGC Interface Standard

OGC 12-100r1 OGC GML 3.2.1 Application Schema – Coverages – GeoTIFF Coverage Encoding Profile, v.1.0 OGC Interface Standard OGC 10-140r1 OGC Web Coverage Service 2.0 Interface Standard - Earth Observation Application Profile

v.1.0.0, 26/02/2014. OGC Interface Standard

Web Processing Service (WPS)

No standards under configuration control

Download Service OGC 13-043, Download service for EO Products, Version 1.0, 31/01/2014. OGC Best Practice Identity Management

OGC 07-118r9, User Management Interfaces for Earth Observation Services, Version 1.1, 28/04/2014. OGC Best Practice

!

OGC ®

OGC Big Data Working Group

•  Open forum on how to use OGC standards for Big Data interoperability, access, and especially analytics – Encourage collaborative development among participants

representing many organizations and communities, – Ensure appropriate liaisons to other Big Data relevant working

groups, both inside and outside OGC.

•  Big Data agenda – September 2014 – GeoWave Introduction and Overview (NGA) – Well Known Text for Accumulo (NGA) – Spatio-Temporal Data Cube Analytics (Jacobs Univ.) – Elasticity for WMS Hosted in a Cloud (Amazon) – NOAA Big Data Partnership RFI (NOAA)


http://www.opengeospatial.org/projects/groups/bigdatadwg

OGC ®

Discrete Global Grid Systems

Slide Source: Matt Purss, Geoscience Australia

ANZLIC National Nested Grid***

SCENZ-Grid by CSRIO for NZ

Earth System Spatial Grid by China

Snyder Grid e.g. PYXIS

*** See BIDS 2014 paper: Iterating Petabyte-scale Earth Observation processes in the Australian Geoscience Data Cube, Simon Oliver (Geoscience Australia)

OGC ®

Discrete Global Grid System (DGGS) Standards Working Group (SWG)

•  Develop common criteria that will define conformant DGGSs – Considering Goodchild criteria

•  Develop conceptual standard to facilitate data fusion between DGGSs using OGC Standards –  to make them interoperable – with

conventional and other DGGS data –  to standardize operations on them

•  Engage stakeholders to encourage new use cases and adoption of interoperability through DGGSs

http://www.opengeospatial.org/projects/groups/dggsswg

OGC ®

Big data in OGC Testbeds

•  OGC Interoperability Program – Prototyping unites users and industry in accelerating interoperability –  85 initiatives since 1999: Testbeds, Pilots, Experiments

•  OGC Testbeds with Big Data from space – OWS-8: Observation Fusion – OWS-9: OWS Innovations – OGC Testbed 10: Open Mobility and Cloud – OGC Testbed 11: Urban Climate Resilience

•  Highlights discussed in the following categories – Maps; Coverages; Performance


OGC ®

Maps and Big Data

•  Web Map Tiling Service (WMTS) – WMTS is WMS tuned for high velocity maps – Serves map tiles with predefined content, extent, and resolution – TileMatrixSet: collection of tile matrices defined at different scales

•  OWS-9 evaluated diverse raster map tiling schemes – WMTS, Tile Map Service (OSGeo), MBTiles, TileCache

(MetaCarta), and various others

•  Recommendations forwarded to OGC Standards Program. – Basis for “WMTS – Simple” based on limited TileMatrixSet – Revised WTMS standard nearing completion in SWG


OWS-9 Map Tiling Methods Harmonization (Editor Joan Maso, OGC Doc 12-157)

OGC ®

WMTS Example: Several sources, Same TileMatrix

Copyright © 2013 Open Geospatial Consortium

a.tile.openstreetmap.org mt1.google.com otile1.mqcdn.com/tiles/1.0.0/map

tile.opencyclemap.org/cycle

ecn.t0.tiles.virtualearth.net

www.ogc.uab.cat/SITiled/GeoEye_Haiti_M

X=9798 Y=14664

X=9799 Y=14664

X=9800 Y=14664

X=9798 Y=14665

X=9799 Y=14665

X=9800 Y=14665

OGC ®

OWS-8 Coverages as Big Data

•  WCS 2.0 Earth Observation Application Profile (EO-AP) –  Build on WCS 1.0 EO-AP developed in ESA’s HMA project.

•  …to include data models for: –  2-D Coverage with latitude and longitude axes, e.g., a hyperspectral satellite scene. –  Subset of a 2-D satellite scene (either radar or multispectral) –  Time series of co-registered 2-D satellite scenes –  Stitched Mosaic as a 2-D horizontal for several co-referenced non-overlapping datasets –  Dataset Series as a collection of coverages;

•  …supporting these data products: –  MODIS –  ENVISAT MERIS L3 demonstration products (http://earth.eo.esa.int/level3/) –  ENVISAT ASAR Wide Swath –  Limb-scanning instruments (e.g. Microwave Limb Sounder) typically produce vertical profiles, –  Narrow-swath instruments (e.g., CALIPSO) produce vertical cross-sections.

•  …and these encodings: –  GeoTIFF, netCDF / CF-netCDF, JPEG2000, HDF-EOS

•  With a WCS 2.0 EO-AP Compliance Test and Reference Implementation


OWS-8 WCS 2.0 EO Application Profile ER (11-096) OWS-8 WCS 2.0 EO Application Profile Compliance Tests and Reference Implementation ER (11-095) OWS-8 ER Metadata Mapping between NASA ECS/HDF-EOS and WCS 2.0 (11-090)

OGC ®

OWS-8 Coverage Processing

•  WPS for EO data analysis of multiple coverages and other data

•  WCPS for advanced access to EO Coverages – multi-coverage fusion

capabilities –  queries taken from real-life

examples


OWS-8 Geoprocessing of EO (Editor P. Baumann, OGC 11-116)

WCPS and WPS Orchestration

OGC ®

OWS-8 Big Data Analytics on Motion Imagery

© 2011 Open Geospatial Consortium, Inc.

Display to Common Operational Picture …and send to Mobile Client

References: Ingo Simonis, IGSi OGC Doc 11-108 R. Cass, M. Simms, Compusult, OGC Doc 11-134

OGC WPS for generating track features STANAG 4676 represented as GML features VMTI Implemented in OWS-8

OGC ®

VMTI to GeoSMS OWS-8 Mobile Client Implementation

•  Android app for OGC GeoSMS, WFS features and KML •  Mobile client receives GeoSMS with URL for WFS track feature

Copyright © 2012, Open Geospatial Consortium

Videos of OWS-8 Moving Object Tracking http://www.opengeospatial.org/pub/www/ows8/tracking.html

OGC ®

OWS-9 Access to Big Data Coverages

•  OWS-9 WCS and OPeNDAP – DAP and WCS compliment each other. – Easy to use DAP to ‘reach into’ files and access their components – WCS provides specialized interface for geospatial data

•  OWS-9 WCS for LIDAR –  Implementation of WCS 2.0 service that serves the LIDAR data in

NITF format –  service supports the JPEG 2000 output format along with GMLJP2

metadata description as well as the JPIP protocol to deliver the output JPEG2000 data.

– Demonstrated WCS with JPIP streaming for GMLJP2.


OWS-9 Coverage Access ER (Editor P. Bauman, OGC 12-095) OWS-9 OWS Innovations WCS for LIDAR Engineering Report (Editor W. Han et.al., OGC 12-155)

OGC ®

OGC Testbed 9: LiDAR Geoprocessing with OGC Standards


OWS-9 OWS Innovations WCS for LIDAR Engineering Report (Editor W. Han et.al., OGC 12-155)

OGC ®

Performance of OGC Services in the Cloud

•  OGC service standards have been widely adopted and deployed across the world but little information has filtered out regarding the performance and scalability of products based on OGC standards.

•  Easy access to large IT computing resources such as the Amazon EC2 Cloud infrastructure provides an opportunity to use a flexible and low cost IT resource environment to investigate the performance and scalability of products based on OGC standards.

•  OGC Testbed 10 tested web mapping and other geo-processing use cases as a way to characterize the performance of OGC data services deployed in Cloud infrastructures.


OGC ®

OGC Testbed 10: WMTS Massively Available with Cloud Services

© 2014 Open Geospatial Consortium

Concurrent user access to OGC-compliant map servers using an Amazon EC2 with an S3 network attached storage configuration

OGC ®

OGC Testbed 10: Performance & Scalability of WMS/WMTS

•  Understand how OGC services perform on clouds through testing in OWS-10 Testbed

•  Response time with concurrent users for – WMS: linear – WMTS: flat to 1800

!

OGC Testbed 10 Performance of OGC Services in the Cloud ER (Editor Edric Keighan, OGC 14-028r1)

OGC ®

OGC Testbed 10: SAR Interferometry with WPS on SBAS Cloud

•  Performance enhancements with Cloud deployment of SBAS (Small Baseline Subset) processing application using WPS and OpenSearch OGC Web Services


!

•  Exploit 64 differential SAR scenes for the generation of time series showing ground displacements over a decade in geological sensitive areas.

•  Part of an ongoing effort from ESA, CNR-IREA and Terradue partners

OGC ®

OGC Testbed 10: Moving Cloud Processing to Data

1.  An OGC WPS Service “bought” on Terradue Cloud Marketplace

2.  then deployed to Amazon UK Cloud

3.  which computes using data from Interoute Cloud UK

Experiments led by Terradue in OGC Testbed 10

OGC Testbed 10 Performance of OGC Services in the Cloud ER (Editor Edric Keighan, OGC 14-028r1)

As a result of this testbed: User organizations able to request service providers (data provider, algorithm provider) to operate under a multi-tenant Cloud environment with specific performance constraints, in order to build and run scientific applications that deliver added-value products.

OGC ®

OGC Testbed 11: Urban Climate Resilience


•  simulation models integration •  high-resolution data •  auxiliary data integration •  data representation •  WFS-T Transactions

•  Desktop/mobile clients •  Geosynchronization •  REST and GeoJSON

•  What-if scenarios & models •  Social/political data (TJS) •  GeoPackage creation •  Georeferenced grid harmon. •  Streaming •  Alerting

•  GeoPackage creation •  GeoPackage synchronization •  Image streaming tests

OGC Testbed 11 UCR Thread Architect: Dr. Ingo Simonis, OGC Director Interoperability and Science

OGC ®

OGC Testbed 11 Threads RFQ/CFP: http://www.opengeospatial.org/standards/requests/126

© 2014, Open Geospatial Consortium 30

Geo4NIEM

• NIEM V3.0 IEPD Geoprocessing Round-Trip

• NIEM Enhancements Security Tagging

• NIEM-GML Feature Processing

• Security: Identification Authorization, Access

• And more…

Urban Climate Resilience (UCR)

• Climate/Big Data processing - WPS

• Hi-Res Flood Model

• Social Media

• WFS-T with REST

• GeoPackage, GeoSync

• Georeferenceable Grid Harmonization

• And more…

Cross-Community Interoperability (CCI)

• Aviation: NOTAM, AFX

• REST and SOAP

• JSON/GeoJSON

• Semantic Broker: Social Media, Linked Data, GeoSPARQL

• SPARQL for Symbology

• Compliance

• And more…

Cross Thread Scenario: Displaced Population Due to Coastal Inundation

OGC ®

Relevant OGC Engineering Reports

These engineering reports (ER) provide more detail on the topics addressed in OGC Testbeds listed above: •  OWS-8 ER Metadata Mapping between NASA-EOS and WCS 2.0 (11-090) •  OWS-8 WCS 2.0 EO Application Profile ER (11-096) •  OWS-8 WCS 2.0 EO AP Compliance Tests and Reference Implementation ER

(11-095) •  OWS-8 Geoprocessing of EO (11-116) •  OWS-9 Map Tiling Methods Harmonization (12-157) •  OWS-9 Data Quality for Web Mapping ER (12-160) •  OWS-9 Coverage Access ER (12-095) •  OGC Testbed 10 Service Integration ER (14-013) •  OGC Testbed 10 Performance of OGC Services in the Cloud ER (14-028r1)


OGC ®

For Details on OGC Standards…

•  OGC Standards – Freely available –  www.opengeospatial.org/standards

•  OGC Reference Model (ORM) – Overview of OGC Standards Baseline – Resource for defining architectures for specific

applications –  www.opengeospatial.org/standards/orm

George Percivall, gpercivall at opengeospatial.org

Technology

Geospatial Temporal Open Standards for Big Data from Space (BiDS2014)