Baking an FME Dashboard: Two cups of Python with a pinch of frustration

Jubal HarpsterPrincipal and Co-Founder Spatial Development International

Thank You!

A little about Me:

Regular FME User Since 1998

Started translating to ArcSDE at version 3.

FME Server Beta User since ~2007

I Use FME to..

Get Features & Checkins from Fousquare

Detect Changes to Open Street Map

Creating Regional Basemaps

The “Dashboard”

The Business Problem in a nutshell Port of Seattle runs multiple FME translations

Need the ability to audit transformations

Need to verify if people are running their workspaces

Need to know what was updated when and by who.

Need to know if any features failed Be able to ‘see’ the results enterprise wide on

Dashboard

The “Dashboard”

Piece of Cake!

Decided early to use TCL script to get windows user name TCLCaller: set x $tcl_platform(user)

Create a simple custom transformer FMX that can easily be ‘dropped’ into each workspace

With some Python on Shutdown Invoke pyfme

The “Dashboard”

The Approach Custom Transformer for flexibility across

workspaces TCL for minor functions

Python Statup and Shutdown Scripts Start counts End counts Status messages Errors etc

The “Dashboard”: Challenge One

Start Up only have access to limited information

Problem: No way to tell FME the destination before it gets there (to get start counts)

Solution: Published Parameters written to the Log

The “Dashboard”: Challenge Two

Shutdown Scripts only have access to limited information

Problem: FME loses generated values (like User Name)

Solution: Track values in the logs and in the DB using SQL

The “Dashboard”

Our elegant & simple solution becomes a dreaded log file parse

The “Dashboard”:Challenge Four

But what about errors? How do we test? How do we record problems or know if FME

crashes?

Problem: If things really go wrong Shutdown won’t execute

Solution: The Terminator

The “Dashboard”:Challenge Five

More Errors Sometimes 3rd party formats kill processes

(i.e. illegal CAD features in ArcSDE)

The Problem: Sometimes bad things happen

The Solution: Set FME To Ignore Errors

The “Dashboard”: The Process

• Insert failures

• Grab Parameters

Startup

• Log Features

• Log Derived Parameters

Mid-way

• Update SuccessShut

down

The “Dashboard”: Using pyfme

Pyfme in Python callers treats input as features Create Geometry Get & Set Attributes

Can Access Published Paramters param_buffwidth = pyfme.macros.BufferWidth

Access Transformers and Piplines

Tips: The integrated IDE is OK. But use something

else. I use Aptana Studio or Eclipse & pydev

Create a class that invokes the pyfme modules

class MyPythonFactory(object): def __init__(self):

pass def input(self,feature):

self.pyoutput(feature) def close(self):

pass

The “Dashboard”: Using pyfme

Using pyfme

Tips: Use Custom Transformers

Embedding PythonCallers in a custom transformer is an easy way to install and distribute functions

There are lots of Python developers out there Python samples at FMEpeida Olivers Python Corner on FMEpedia Python.org

@FMEDoctors on twitter can help in a pinch But only if the problem can be solved in 140 chars

Thank You!

Questions?

For more information: Jubal Harpster

jharpster@spatialdev.com @jharpster

Spatial Development International http://www.spatialdev.com