Embed Size (px)
Citation preview
Managing Data Interoperability with FME
Tony KentApplications Engineer
IMGS
IMGS
We deliver innovative spatial solutions
For the desktop, web and mobile
Built on our partner’s technology
Designed to meet the challenges of Government, Mapping Agencies, and Utility & Communications Customers
Safe Software
Powering the flow of spatial data with FME
3
Enabling people to use their spatial data where, when and how they want to
Most Used Spatial Interoperable Solution in Ireland
Why Spatial ETL?
Significant proliferation of different spatial data formats and types
Hundreds of formats, with more added each yearMultiple types of data stored in multiple systemsUnique data model requirements for each application
4
Why Spatial ETL?
Traditional approaches to data translation and data model manipulation are not viable
Complex, inefficient and time-consuming
5
Why Spatial ETL?Increasing pressure for access to spatial data
More users, beyond traditional GIS usersExpectations of real-time custom data views, 24x7
6
FME Capabilities
7
The only complete spatial ETL solution
Translate spatial data from one format to another
Transform spatial data into the precise data model you need
Integrate different data types into a single data model
Distribute spatial data to users where, when and how they need it
FME Desktop
Flexible and powerful spatial ETL toolset
Translate, transform and integrate data in hundreds of formats
Graphical authoring environment
8
Step 1 - Extract
Select and add the source dataset(s)
Step 2 - Transform
Add transformers to manipulate the
data as it moves from source to
destination Step 3 – Load
Load the transformed data into a
destination format and source
FME Workbench
9
Use simple point and click to easily define spatial data flows to translate, transform and integrate your data
Examples
Automating Ordnance Survey data updatesPushing NTF data to multiple GIS platforms
Stripping out unnecessary data
Adding custom styling and symbology – CAD
E.g. Eircom, ESB, Fingal County Council
Publishing data to internal public portals Bulk and transactional updates
Fire wall Friendly – use selected port
Completely automated
E.g. Dublin City Council
Open Data Challenge
You want to meet Open data requirements, but your data is organized rather differently
?
?
What FME does …
Build data bridges to your SDI
SDI Harmonization Core Concepts
Harmonization: implied requirement for building an SDI
Disparate sources must be mapped to a common destination data model
Core to the harmonization workflow is a process called schema mapping.
Delivered by services based on open standards
Harmonization Principles
Typical stages:
1. Evaluation
2. Assembly
3. Transformation
4. Validation
5. Publication
Based on the Spatial ETL concept (Extract, Transform and Load), as applied to INSPIRE SDI’s
Metadata – Data about data
Describes data structurestables
geometry types
data types
fields
Describes data contentcoordinate system
extent
modification date
quality, ownership, etc.
Metadata - Purpose
Key FME Metadata Capabilities
Reading
Writing
Updating
Harvesting
Validating
Integration with web services
Data Transformation - Schema
Reshape source data to match required destination schema
Schema mappingfeature type
attribute name
new attribute creation
code lists
conditional value
mappings
Feature Type Mapping in FME Workbench
Attribute Mapping in FME Workbench
Schema Mapping in FME
Value Mapping
FME Data Model Restructuring: Attribute Names & Values
FME SchemaMapper: INSPIRE geographic names
Name mapping
Name & value mapping
FME Workspace
Transformation: Geometry
Non-spatial to spatial
Geometry extraction (spatial to GML)
Representation transform: CAD drawing lines with labels to GIS polygonal features with attributes
Coordinate System Reprojection (ED50 to ETRF89)
Simple to complex geometrySource point and polygon data to multiple geometric representations (city as point / area, river as line / area)
Generalization and interpolationHighly granular national and regional datasets often require thinning to be usable on pan-European scales
Validation
Schema validation i.e. INSPIRE (xsds)
Data integrityUnique IDs
Geometric integrity (closed polygons)
Null values (nullable?)
Valid values: ranges and domain codes
Data gaps
Bounds
Network integrity
Custom validity rules specific to domain
Validation automation via FME Server upload
Ensure data quality throughout the data transformation process
Publish workspace to FME ServerStore the workspace in a central repository
Make your FME workspaces available to others –over the web
Register the workspace with one or more services (Data Streaming, Data Download, etc.)
Publication with FME Server
24
Format translation
Schema mapping
String and list manipulation
Data validation
Database load and extract
XML,GML,WFS: reading, validation, publication
Web services: WFS, WMS, integration with others
Metadata support
Enterprise services with FME Server
FME Tools for INSPIRE
FME can provide all the tools to help build support your data sharing needs:
Integrate your data sources
Manage your meta data catalogues
Transform your data to standard schemas
Publish the data in the required formats
Summary
