FME Process Optimisation at the Ordnance Survey Great Britain An exercise in best practice

david.eagle@1spatial.com Principal Consultant FME Certified Professional & Trainer 1Spatial

Agenda

•  The Ordnance Survey

•  The Object Editor Adapter

•  Project Goals

•  FME as a Test Harness

•  Deploying ‘Best Practice’ in FME •  Tips and tricks

•  Project Benefits

2  

Ordnance Survey - Then

3  

•  The original ‘Board of Ordnance’, MOD of the day

•  Started survey work in the late 1700s

•  Comprehensively mapped the south coast to be able to effectively repel invasion

•  http://goo.gl/02tVFZ

Lands End, Mount's Bay, Cornwall

*

Ordnance Survey - Now

4  

•  Self financing part of the UK government

•  £120-million-a-year civilian organisation

•  At the forefront of the digital economy

•  Remit covers Great Britain (England, Scotland and Wales)

•  243,241 sq km / 93,956 sq miles

•  Integrated data capture for field surveyors, HQ-based editors, and external suppliers

*

Geospatial Data Management System (GDMS)

•  Seamless data maintenance solution

•  Intelligent and integrated maintenance data model

•  Allows separation of product data

•  Includes automated data validation

•  Platform for new product development

•  650+ data editors

•  500+ million features

•  4,000 data maintenance jobs per day

*

GDMS – Production

*

GDMS – Production Architecture

•  Includes best of breed COTS components from •  1Spatial •  BAE Systems •  ESRI •  Intergraph •  Microsoft •  Oracle •  Safe Software •  Snowflake Software

•  Scalable infrastructure

•  Disconnected editing thanks to the ‘Object Editor Adapter’

*

Object Editor Adapter

8  

•  Transforms data into a structure usable in the Object Editor map application

•  FME underpins the transformation

•  Schema is manipulated into ‘understandable’ structure from the database

GML OEA (FME) PGDB

*

Project Goals

9  

1.  Document process and make maintenance practical

•  Schema changes pending

•  Component needed to be better understood

2.  Upgrade process to FME 2013 (look beyond 2013)

•  Built with FME 2009

•  Utilise recent FME enhancements

3.  Improve efficiency & performance

•  Remove redundancy

“If it ain’t broke…”

10  

•  Huge amount of change in FME since 2009

•  Attribute management particular interest

•  OS needed to unpick the workspace to be able to update it anyway

•  Workspace opening time was slow, impacting batch run time and FME Server

–  3 minutes

–  c.60mb (most *.fmw don’t exceed a few 100kb)

•  Quite a few transformers…

Transformers, transformers everywhere

11  

•  943 –  Custom Transformer(s) = 255

–  AttributeRenamer = 211

–  Tester = 150

–  AttributeRemover = 87

–  AttributeCreator = 51

–  ListExploder = 72

–  CaseChanger = 21

–  ListElementCounter = 18

–  AttributeSplitter = 12

–  AttributeSetter = 10

–  ListIndexer = 9

–  AttributeCopier = 8

What is a ‘Test Harness’?

12  

In software testing, a test harness or automated test framework

is a collection of software and test data

configured to test a program unit

by running it under varying conditions

and monitoring its behaviour and outputs.

Wikipedia

Using FME as a ‘Test Harness’

13  

•  The process had to change but the data must not!

•  FME to check the old process against the new

•  Controlled via a spreadsheet, a WorkspaceRunner and Published Parameters

GML  

PGDB  

OLD

PGDB  

NEW

Detect Change

Failures: Spreadsheets and

Geometry *

14  

Dynamic Schema

15  

•  Introduced around FME 2009

–  Existing process didn’t use it

•  Enable ‘Dynamic’ schema mode

–  Whatever schema is read, gets written

–  Decouple schema from format

•  http://blog.safe.com/2010/02/fmeevangelist54/

Dynamic Schema

16  

•  96 input layers can be reduced to 1!

–  Magic Transformer = FeatureTypeExtractor

–  Workspace infinitely less daunting

Reduce Transformer Duplication

17  

•  Helped by Dynamic Schema Mode

•  Try not to duplicate operations

Reduce Transformer Duplication

18  

•  ‘Like’ operations carried out on all features

–  Filter features off to unique processes when needed

•  20% of data had a ‘Level’ –  21 CaseChangers used to apply UPPERCASE

–  Replaced with 1 StringCaseChanger

•  69 Custom Transformers with embedded Python used to format Date attributes

–  All features passed through a StringReplacer to remove hyphens

•  2013-01-01 to 20130101

Reduce Transformer Duplication

19  

Custom Transformers

20  

•  What?

•  A container for many regular transformers

•  Allows a common set of tasks to be reused in 1 or many workspaces

•  When exported becomes an *.FMX file for sharing

•  The FME Store was introduced in 2012 –  2014 FME Store mature with 130+ transformers (and other resources)

•  http://fmestore.safe.com •  http://fmepedia.safe.com/articles/Samples_and_Demos/Custom-Transformer-

KMLDiagrammer

•  It’s increasingly likely your workspaces will contain Store resources

•  FME now auto connects & downloads any resources required

Custom Transformers

21  

•  Support for versioning •  Ports are not disconnected when renamed

–  http://blog.safe.com/2013/12/fmeevangelist121/

•  Embed/link all instances of a given transformer •  Manage input and output schema on Custom Transformers

Custom Transformers 2014

22  

•  14 unique custom transformers used 255 times

•  PGDB had constraints, many tables did not allow null values –  Custom transformers tested and removed null attributes –  Thanks to Generic mode expose constrained attributes and pass through a

PythonCaller to remove the null attributes

•  FME 2014

Use Custom Transformers Effectively

23  

Schema Mapping

24  

•  Attribute names are hardcoded into 211 AttributeRenamers in the workspace

–  Changes to target schema means attribute hunting!

–  Biggest challenge for ongoing maintenance

•  1x SchemaMapper replaces all AttributeRenamers

–  Non-FME user can alter the process by editing

AttributeMapping.csv

Documentation

25  

•  Always document your Workspace

•  The OEA process was documented

–  But didn’t include detailed explanations ‘why’

•  You should be able to read a Workspace and understand it

–  Highlight dependencies and use XML notation for naming

•  In 2014 canvas annotations taken to the next level

Documentation – Self contained

26  

Before and After

27  

Benefits

28  

Before After Improvement Time to open (from local disk) 2mins 12secs 31secs 1min 41secs File size 56mb 13.8mb 75% reduction Lines of machine written code 450,332 111,334 75% reduction Number of transformers 943 336 64% fewer Sample run time 1hr 7mins 48mins 19mins

•  Current, documented and ‘owned’ by the Ordnance Survey

•  Robust production extraction routine

•  Simpler to maintain and change

–  Some schema changes possible without opening FME!

29  

david.eagle@1spatial.com Principal Consultant FME Certified Professional & Trainer 1Spatial