Using FME to support Open Data initiatives and INSPIRECiaran KirkTechnical Director

6th September 2012

Tony KentGIS Technician

The Challenge

You want to meet Open data requirements, but your data is organized rather differently

?

?What we do …

We build data bridges – to your SDI

Harmonization Core Concepts

Harmonization: implied requirement for building an SDI

Disparate sources must be mapped to a common destination data model

Core to the harmonization workflow is a process called schema mapping.

Delivered by services based on open standards

INSPIRE Harmonization Principles

Typical stages:1. Evaluation2. Assembly3. Transformation4. Validation5. Publication

Based on the Spatial ETL concept (Extract, Transform and Load), as applied to INSPIRE SDI’s

Evaluation

Assess destination schema and data requirements Assess source datasets and schema Consider fundamental differences in representation,

resolution Closely inspect actual representative datasets

Data Assembly

Assess the diversity of source data types: vector, raster, CAD, GIS, database, text, XML, web,

3D, sensor and non-spatial Review format and semantic translation needs Decide how to perform necessary joins

ID joins, spatial relates, nearest neighbor, one to many relationships

Goal is to build a data structure that corresponds with your published standard

Metadata – Data about data

Describes data structures• tables• geometry types• data types• fields

Describes data content• coordinate system• extent• modification date• quality, ownership, etc.

Metadata - Purpose

Key FME Metadata Capabilities

• Reading

• Writing

• Updating

• Harvesting

• Validating

• Integration with web services

CSW: Catalog Service for the Web

• Deegree

• GeoNetwork

• OpenGIS

• TerraCatalog

Metadata Services: CSW

• Reading• Writing / Publishing• Updating

Data Transformation - Schema

Reshape source data to match required destination schema

Schema mapping feature type attribute name new attribute creation code lists conditional value mappings

Schema Mapping in FME

Feature Type Mapping in FME Workbench

Attribute Mapping in FME Workbench

Schema Mapping in FME

Value Mapping

FME Data Model Restructuring: Attribute Names & Values

FME SchemaMapper: INSPIRE geographic names

Name mapping

Name & value mapping

FME Workspace

Transformation: Geometry

Non-spatial to spatial Geometry extraction (spatial to GML) Representation transform: CAD drawing lines with labels

to GIS polygonal features with attributes Coordinate System Reprojection (ED50 to ETRF89) Simple to complex geometry

Source point and polygon data to multiple geometric representations (city as point / area, river as line / area)

Generalization and interpolation Highly granular national and regional datasets often require

thinning to be usable on pan-European scales

Validation

Schema validation i.e. INSPIRE (xsds) Data integrity

Unique IDs Geometric integrity (closed polygons) Null values (nullable?) Valid values: ranges and domain codes Data gaps Bounds Network integrity

Custom validity rules specific to domain Validation automation via FME Server upload

Ensure data quality throughout the data transformation process.

Publication

Produce INSPIRE compliant GML Provide discovery, view or download services, for

WxS, GML and other desired formats Publish with FME Server or integrate with your

geo web server of choice Spatial Data Services

Publish workspace to FME Server Store the workspace in a central repository

Make your FME workspaces available to others –over the web

Register the workspace with one or more services (Data Streaming, Data Download, etc.)

Publication with FME Server

21

Open Data Demonstration

Format translation Schema mapping String and list manipulation Data validation Database load and extract XML,GML,WFS: reading, validation,

publication Web services: WFS, WMS, integration

with others Metadata support Enterprise services with FME Server

FME Tools for INSPIRE

FME can provide all the tools to help you support your data sharing needs:

Integrate your data sources Manage your meta data catalogues Transform your data to standard schemas Publish the data in the required formats

Summary

Thank You!

Questions?

For more information: Ciaran Kirk – ckirk@imgs.ie Tony Kent – tkent@imgs.ie

support@imgs.ie