Embed Size (px)
DESCRIPTION
Getting the best performance out of FME is as important to us as it is to you. In this webinar, you’ll get tips from three FME experts - Mark Ireland (FME Evangelist), David Eagle (FME Certified Trainer and Professional) and Dale Lutz (Safe Software Co-founder). They’ll share easy-to-apply advice on: querying databases efficiently, making the most of FME's new multiprocessing capabilities, and simple techniques to speed up your workflows.
Citation preview
Turbocharging FME: How to Improve the Performance of Your FME Workspaces
Host:Ken BraggPanel: Mark Ireland, David Eagle, Dale Lutz
December 12, 2012
12/12/12
Host
Ken BraggEuropean Services ManagerSafe Software
Questions are Encouraged!
Our Panel
Mark Ireland (iMark)Product Evangelist, Safe Software
Killarney, Manitoba, Canada
David A. Eagle (The Eagle)Principal Consultant, 1Spatial Group
Ltd.Cambridge, United Kingdom
Dale Lutz (The Founder)Co-Founder & VP of Development Safe SoftwareVancouver, BC, Canada
Secret HQ: Vancouver
Thousands of happy customers, certified professionals, trainers, system integrators, and value added resellers; all around the globe!
100 exuberant employees
FME Capabilities
Transform Data to Use and Share
Convert spatial data between hundreds of formats
Transform spatial data into the precise data model you need
Integrate multiple different data types into a single data model
Share spatial data with people where, when and how they need it
FME Workbench: Graphical Data Flow Authoring
FME : Our PromiseFaster than a Helicopter? Faster than ever!
Turbocharge your Workspaces!
Poll
How Long do your Workspaces Take?
iMark – Mark Ireland
Workbench Performance
Workspace Performance
The key to workspace performance is to reduce the amount of data being processed.
Feature vs Group
Some transformers work on one feature at a time; others work on a group of features.
Feature Based
Measuring individual heights...
5’ 9”5’ 7”5’ 8”
Feature-based transformers work on one feature at a time. They are LESS resource intensive.
Group Based
Measuring average heights for each gender...
5’ 9”5’ 8”
6’ 0”
5’ 7”
Group-based transformers work on many features at a time. They are MORE resource
intensive.
Average = 5’ 9”
TransformersSome group-based transformers have settings to turn them into feature-based
• AttributeAccumulator• Aggregator• Clipper• NeighborFinder
First WriterOrder your writers so the one receiving the largest amount of data is first in the list.
http://evangelism.safe.com/fmeevangelist64-2/
Attribute Handling
• Remove unwanted attributes• Beware of large lists
And don’t store geometry as an attribute!
Poll
Where does it hurt?
The Eagle – David Eagle
Working with Databases
Turbocharging FME: Working with Databases
David EaglePrincipal ConsultantFME Certified Professional & Trainer1Spatial Group Ltd
December 12, 2012
Learn to be a ‘Speed Reader’
Being able to read quickly is often beneficial
When you have a database, you often have volume Read data intelligently and achieve efficiency!
A typical approach is where FME does the heavy lifting
1 - Read_and_clip.fmw
Master FME ‘Parameters’ - become an FME Hero! Parameters allow you to apply control Enable flexible workspaces – Prompt and Run Make the database do the work
Reader Parameters
Embrace the Spatial Index
Spatial indices are used by spatial databases (databases which store information related to objects in space) to optimize spatial queries. Many conventional index types do not efficiently handle features such as how far two points differ, or whether points fall within a spatial area of interest.
http://en.wikipedia.org/wiki/Spatial_database
Read ‘within’
All Readers since FME 2011 have allowed you to specify a reader bounds
Not all formats have a true spatial index No performance gain but less data in memory
Spatial databases can have a spatial index Create index with SQL or create with FME!
2 - Read_with_spatial_index.fmw
Read ‘where’ and ‘what’
Use FME to issue a SQL Where Clause Be specific about the features you need www.w3schools.com/sql beginners resource
Leverage the power of the parameter ‘Feature Types to Read’ Always see what’s new
3 - Read_WhereandWhat.fmw
Synchronous Reading/Writing
Mid translation reading/writing may be required Beneficial when you want to be specific Allows native format query – greater efficiency
SynchronousReading.fmw
Ask your friendly DBA
Prebuild the data you want as a ‘View’ A stored query that forms a virtual table
More efficient and dynamic for repetitive queries FME treats Views like normal Tables Allows you simplify your Workspace
Materialized Views (Oracle, DB2, MS SQL Server, PostgreSQL)
Results of a Query stored on disk
Database Joins
There’s lots of options for carrying out Joins (not limited to databases) Joiner vs FeatureMerger
http://evangelism.safe.com/fmeevangelist79/ FME 2012 introduced the InlineQuerier
Cache data to disk mid-process (SQLite) Write SQL against non-database formats http://evangelism.safe.com/fmeevangelist97/
Transformation Challenges
For more information, contact: [email protected] +44 (0)1223 420414 @david_eagle @1spatial
Web resource: www.1spatial.com/fme
Over to you Dale!
Thank You!
The Founder – Dale Lutz
Multiprocessing - Faster than Ever
Multiprocessing: Definition
Multiprocessing: Definition
Multiprocessing is the use of two or more central processing units
(CPUs) within a single computer system
- Wikipedia
Multiprocessing: Motivation
Multi-Core and HyperThreaded CPUs Very common Potential to increase compute-bound throughput
Multi-Processing:Sample Performance Boosts
3D Clipping of Point Clouds 2 X faster
LineJoiner and Buffer 2 X faster
Raster Cell Coercer/Point on Raster 4 X faster
Multiprocessing & FME
Option on Many Transformers Multiprocesses Groups
Multiprocessing & FME
Multiprocessing & FME
Option on All Custom Transformers
Multiprocessing: Levels Of Parallelism
None Minimal Moderate Aggressive Extreme
Multiprocessing: Levels Of Parallelism
Parameter Processes
No Parallelism
1
Minimal Cores / 2
Moderate Cores
Aggressive Cores * 1.5
Extreme Cores * 2
License Cap
Base 4
Professional 8
Other 16
Multiprocessing: Summary
New in FME 2012
Leverages Multi-Core and HyperThreading CPUs
Controlled by: Levels of Parallelism Group-By
Can give significant performance boost http://fmeevangelist.com/fmeevangelist94 http://fme.ly/parallel (FMEpedia Article)
Multiprocessing
DEMO
Share Today’s Webinar
Today’s webinar was recorded View our archive at:
http://fme.ly/archive
APRIL
Special Offer!
Win Free Training!
View the offerings at: http://safe.com/training
For more information contact: Safe Partners
http://www.safe.com/partners/find-a-sales-partner Safe Sales:
[email protected] www.FMEPedia.com www.safe.com/support
Dale@DaleAtSafe
Mark@FMEEvangelist
David@david_eagle
Ken@KenatSafe
Thank You!
