Turbocharging FME: How to Improve the Performance of Your FME Workspaces

Turbocharging FME: How to Improve the Performance of Your FME Workspaces

Host:Ken BraggPanel: Mark Ireland, David Eagle, Dale Lutz

December 12, 2012

12/12/12

Host

Ken BraggEuropean Services ManagerSafe Software

Questions are Encouraged!

Our Panel

Mark Ireland (iMark)Product Evangelist, Safe Software

Killarney, Manitoba, Canada

David A. Eagle (The Eagle)Principal Consultant, 1Spatial Group

Ltd.Cambridge, United Kingdom

Dale Lutz (The Founder)Co-Founder & VP of Development Safe SoftwareVancouver, BC, Canada

Secret HQ: Vancouver

Thousands of happy customers, certified professionals, trainers, system integrators, and value added resellers; all around the globe!

100 exuberant employees

FME Capabilities

Transform Data to Use and Share

Convert spatial data between hundreds of formats

Transform spatial data into the precise data model you need

Integrate multiple different data types into a single data model

Share spatial data with people where, when and how they need it

FME Workbench: Graphical Data Flow Authoring

FME : Our PromiseFaster than a Helicopter? Faster than ever!

Turbocharge your Workspaces!

Poll

How Long do your Workspaces Take?

iMark – Mark Ireland

Workbench Performance

Workspace Performance

The key to workspace performance is to reduce the amount of data being processed.

Feature vs Group

Some transformers work on one feature at a time; others work on a group of features.

Feature Based

Measuring individual heights...

5’ 9”5’ 7”5’ 8”

Feature-based transformers work on one feature at a time. They are LESS resource intensive.

Group Based

Measuring average heights for each gender...

5’ 9”5’ 8”

6’ 0”

5’ 7”

Group-based transformers work on many features at a time. They are MORE resource

intensive.

Average = 5’ 9”

TransformersSome group-based transformers have settings to turn them into feature-based

• AttributeAccumulator• Aggregator• Clipper• NeighborFinder

First WriterOrder your writers so the one receiving the largest amount of data is first in the list.

http://evangelism.safe.com/fmeevangelist64-2/

Attribute Handling

• Remove unwanted attributes• Beware of large lists

And don’t store geometry as an attribute!

Poll

Where does it hurt?

The Eagle – David Eagle

Working with Databases

Turbocharging FME: Working with Databases

David EaglePrincipal ConsultantFME Certified Professional & Trainer1Spatial Group Ltd

December 12, 2012

Learn to be a ‘Speed Reader’

Being able to read quickly is often beneficial

When you have a database, you often have volume Read data intelligently and achieve efficiency!

A typical approach is where FME does the heavy lifting

1 - Read_and_clip.fmw

Master FME ‘Parameters’ - become an FME Hero! Parameters allow you to apply control Enable flexible workspaces – Prompt and Run Make the database do the work

Reader Parameters

Embrace the Spatial Index

Spatial indices are used by spatial databases (databases which store information related to objects in space) to optimize spatial queries. Many conventional index types do not efficiently handle features such as how far two points differ, or whether points fall within a spatial area of interest.

http://en.wikipedia.org/wiki/Spatial_database

Read ‘within’

All Readers since FME 2011 have allowed you to specify a reader bounds

Not all formats have a true spatial index No performance gain but less data in memory

Spatial databases can have a spatial index Create index with SQL or create with FME!

2 - Read_with_spatial_index.fmw

Read ‘where’ and ‘what’

Use FME to issue a SQL Where Clause Be specific about the features you need www.w3schools.com/sql beginners resource

Leverage the power of the parameter ‘Feature Types to Read’ Always see what’s new

3 - Read_WhereandWhat.fmw

Synchronous Reading/Writing

Mid translation reading/writing may be required Beneficial when you want to be specific Allows native format query – greater efficiency

SynchronousReading.fmw

Ask your friendly DBA

Prebuild the data you want as a ‘View’ A stored query that forms a virtual table

More efficient and dynamic for repetitive queries FME treats Views like normal Tables Allows you simplify your Workspace

Materialized Views (Oracle, DB2, MS SQL Server, PostgreSQL)

Results of a Query stored on disk

Database Joins

There’s lots of options for carrying out Joins (not limited to databases) Joiner vs FeatureMerger

http://evangelism.safe.com/fmeevangelist79/ FME 2012 introduced the InlineQuerier

Cache data to disk mid-process (SQLite) Write SQL against non-database formats http://evangelism.safe.com/fmeevangelist97/

Transformation Challenges

For more information, contact: fme@1spatial.com +44 (0)1223 420414 @david_eagle @1spatial

Web resource: www.1spatial.com/fme

Over to you Dale!

Thank You!

The Founder – Dale Lutz

Multiprocessing - Faster than Ever

Multiprocessing: Definition

Multiprocessing: Definition

Multiprocessing is the use of two or more central processing units

(CPUs) within a single computer system

- Wikipedia

Multiprocessing: Motivation

Multi-Core and HyperThreaded CPUs Very common Potential to increase compute-bound throughput

Multi-Processing:Sample Performance Boosts

3D Clipping of Point Clouds 2 X faster

LineJoiner and Buffer 2 X faster

Raster Cell Coercer/Point on Raster 4 X faster

Multiprocessing & FME

Option on Many Transformers Multiprocesses Groups

Multiprocessing & FME

Multiprocessing & FME

Option on All Custom Transformers

Multiprocessing: Levels Of Parallelism

None Minimal Moderate Aggressive Extreme

Multiprocessing: Levels Of Parallelism

Parameter Processes

No Parallelism

1

Minimal Cores / 2

Moderate Cores

Aggressive Cores * 1.5

Extreme Cores * 2

License Cap

Base 4

Professional 8

Other 16

Multiprocessing: Summary

New in FME 2012

Leverages Multi-Core and HyperThreading CPUs

Controlled by: Levels of Parallelism Group-By

Can give significant performance boost http://fmeevangelist.com/fmeevangelist94 http://fme.ly/parallel (FMEpedia Article)

Multiprocessing

DEMO

Share Today’s Webinar

Today’s webinar was recorded View our archive at:

http://fme.ly/archive

APRIL

Special Offer!

Win Free Training!

View the offerings at: http://safe.com/training

For more information contact: Safe Partners

http://www.safe.com/partners/find-a-sales-partner Safe Sales:

sales@safe.com www.FMEPedia.com www.safe.com/support

Dale@DaleAtSafe

Mark@FMEEvangelist

David@david_eagle

Ken@KenatSafe

Thank You!