FME Server Workspace Patterns - ContinuedReed WhittingtonBurns & McDonnell

About Us

FME Certified Pro ServicesBurns & McDonnell, Dallas, TX

Reed Whittington

Burns & McDonnell is a full-service engineering, architecture, construction, environmental and consulting solutions firm.

Our multidiscipline staff of more than 5,000 employee-owners includes engineers, architects, construction experts, planners, estimators, economists, technicians and scientists, representing virtually all design disciplines.

Business Technology Services Burns and McDonnell Technology Consulting Services is a virtual team of technical experts around the globe offering a wide array of services and deliverables including:• Web Sites/Information Management Portals/Doc Management solutions• 2d/Web Maps/3D/4D Globe Applications• Field Data Collection/Inspections/Real-Time Monitoring• CAD/GIS/Database Integration• Automated Report Generation and Distribution• Project Metrics/Analytics/KPI Dashboards

Integrated, field tested, project proven solutions

Many of these services are bundled under OneTouchPM which is a hybrid cloud based, web enabled, managed service allowing project managers to engage their customers and share information, summary progressing with aggregated executive views from all disciplines.

What is a pattern?

In software engineering, a software design pattern is a general reusable solution to a commonly occurring problem within a given context in software design. It is a description or template for how to solve a problem that can be used in many different situations. Not a specific implementation; design patterns are formalized best practices that the programmer can use to solve common problems when designing an application or system.https://en.wikipedia.org/wiki/Software_design_pattern

4 FME Server Workspace Patterns• Workflow Submitter Iterator (or Work Status)

• Strategy for Asynchronous processing of large datasets• Runner / Controller Workspace (or Tool Chain)

• General approach using FMEServerJobSubmitter for workflow control

• Dataset Tool Switch (or Multi-Pass)• Strategy for Synchronous Sequential processing• Operates on same dataset with different modes of operation

• Flatten, Explode, Export (or Import/Merge/Publish)• 3 stages of processing typically used for master data management

Submitter Iterator with Status• Uses a selection strategy to select mutually exclusive

parts of a source data set so that the parts can be operated on independently and run on different fme engines.

• Used to process files or db operations so that sql select statements can be used.

• Needs a better name• Work Status, Delegator, …

• This was the subject of a previous presentation.• http://cdn.safe.com/events/FME2011WorldTour/Austin_Reed_Whittington.pdf

Example using Path reader to Iterate and track status.can also use url as path or list of SQL queries

Submitter Iterator with StatusWait=No

Runner / Controller Workspace• General strategy for Job Control and Tool Chain• Lets you run the controller on desktop• FME 2015 improves FMEServerJobsubmitter:

• In Engine Sequential Run with Wait=Yes• Output datasets to temp files or as specified

Excellent Article by DeWet@Safe:https://knowledge.safe.com/articles/1439/fme-server-workflow-management-1.html

Dataset Tool Switch (aka multi-pass)• The “Multi-pass" pattern presents a common interface

to iterate over datasets using successive refinement. • Meant to be run multiple times with different RUN_MODEs• RUN_MODE Choice defines the Switch behavior• TestFilter on RUN_MODE controls Feature RoutingUseful for

processing directory/tree of files and/or database exports or publishing.

• Interface includes:• SourceDir, StagingDir, Run Modes, DB connection, Publish

location

Use the Path reader to Iterate and Route based on Run Mode

Flatten, Explode, Export• 3 separate workspaces that act together but run asynchronous• Each may use “multi-pass”• Flatten

• Denormalization into one big table• Pack attributes into dictionaries• Think NoSQL and Operational Data Store

• Explode• Unpacks big table• Data model mapping and normalization happens here• May use templates for standard data models• Might also use Dynamic Schema

Flatten, Explode, Export (cont)• Export• Database to files (.kmz, .csv, .json, .gdb, …)• Heavy use of Fanout (ds and ft)• May use Tool Switch RUN_MODE to define

export type or staging/publish processing steps

More possibilities with FME 2016…What do you use?

Tool Chain Workspace• FeatureReader

– Selective reading/caching– Error Checking

• FeatureWriter– Write then read back in

same .fmw– In workspace tool chain– Removes the need for temp

output files of the FMEServerJobSubmitter

Before FME 2016• Required chaining

FMEJobSubmitters together

• Next in chain reads output of Previous

Main point: FME Server Patterns help you develop and organize your data management tools.

Thank you!Reed WhittingtonFME Certified Pro ServicesBurns & McDonnellDallas, TXrwhittington@burnsmcd.com