Sep. 21-22, 2006 v FME Worldwide User Conference - Vancouver Using FME Objects Python API to Provide Platform Independent Translations Andrew Smith, Lagen

Sep. 21-22, 2006v FME Worldwide User Conference - Vancouver

Using FME Objects Python API to Provide Platform Independent TranslationsAndrew Smith, Lagen Spatial, Australia

FME Objects Python API 2

Using FME Objects Python API to Provide Platform Independent

Translations

Andrew SmithSpatial Technical Consultant

Lagen Spatial - Australia


Using FME Objects Python API to Provide

Platform Independent Translations

Introduction Who am I? Project Overview

Python and FMEObjects Method for Development Pitfalls Tips and Tricks

Output ExamplesQuestions




Who am I?

Name: Andrew Smith

Company: Lagen Spatial Pty Ltd

Official Title: Spatial Technical Consultant

Email:[email protected]




Project Overview

Geoscience Australia

plays a critical role by producing first-class geoscientific information and knowledge. This can enable the government and the community to make informed decisions about the exploration of resources, the management of the environment, the safety of critical infrastructure and the resultant wellbeing of all Australians.

www.ga.gov.au




Project Overview

Geographic Data Management Division Solution Name - Geoflex Project Leader - Tony Hunter

Formats Supported

ESRI Shape File (.shp, .shx, .dxf)

Personal Geodatabase

Mapinfo Native Format (.tab), Mapinfo Data Interchange Format (.mif), and

Geographic Mark-up Language (.gml)




Project Overview

Requirements

Be platform independent i.e. the core extraction process is able to execute on both Windows and Unix Operating Systems

Allow a User to Select the Feature Dataset to be extracted

Allow a User to Extract the datasets to on 3 scales National State/Province Map Tile

Custom Symbology Developed by GA to be applied to the Output Datasets

Dynamically update the list of Feature Datasets that are available for extraction through interrogating the underlining Spatial DBMS schema




Platform Independent Translations with FMEObject for Python

Method

FME Workbench to Develop the Translation Process







Copy the Transformers into a text editor to get the FME Script that defines Translation


ArcSDEQuerier

MACRO wheremac SDE_SEARCH_METHOD_FIELD __SEARCH_METHOD

INCLUDE [ if { {SDE_AREA_INTERSECT} == {SDE_NONE} } { puts {MACRO wheremac}} ]

FACTORY_DEF * TeeFactory \ FACTORY_NAME ARCSDEQUERIER_Globber \ INPUT FEATURE_TYPE clip_area_0 \ OUTPUT FEATURE_TYPE __GLOBBED__

FACTORY_DEF * SDE30QueryFactory \FACTORY_NAME ARCSDEQUERIER_Querier \ INPUT FEATURE_TYPE __GLOBBED__ \ @SupplyAttributes(__SERVER,lagen-pc-

1,__INSTANCE,esri_sde,__DATASET,sde,__USERID,sde,__PASSWORD,sdeuser,__TARGET_TABLE,STONNING_CAD,__SEARCH_METHOD,SDE_AREA_INTERSECT,__WHERE_CLAUSE,"",__VERSION,SDE.DEFAULT) \

SDE_SERVER_FIELD __SERVER \ SDE_INSTANCE_FIELD __INSTANCE \ SDE_DATASET_FIELD __DATASET \ SDE_USERID_FIELD __USERID \ SDE_PASSWORD_FIELD __PASSWORD \ SDE_VERSION_NAME_FIELD __VERSION \ SDE_TARGET_TABLE_FIELD __TARGET_TABLE \ $(wheremac) \ WHERE_CLAUSE_FIELD __WHERE_CLAUSE \ QUERY_MODE QUERY \ SEARCH_ORDER SPATIAL_FIRST \ COMBINE_ATTRIBUTES ADD \ OUTPUT_DUPLICATES NO \ GET_SPATIAL_RELATIONS \ OUTPUT RESULT FEATURE_TYPE * \

@RemoveAttributes(__SERVER,__INSTANCE,__DATASET,__USERID,__PASSWORD,__TARGET_TABLE,__SEARCH_METHOD,__WHERE_CLAUSE,__VERSION) \

@SupplyAttributes(_table_name,@FeatureType()) \ @FeatureType(@Concatenate(ARCSDEQUERIER,"_",@FeatureType())) \ @Transform(SDE30,FME_GENERIC)


ClippingFactory

FACTORY_DEF * ClippingFactory \ FACTORY_NAME CLIPPER \ INPUT CLIPPER FEATURE_TYPE clip_area_1 \ INPUT CLIPPEE FEATURE_TYPE ARCSDEQUERIER_STONNING_CAD \ MULTICLIP YES \ CLIPPEE_ON_BOUNDARY INSIDE \ MERGE_CLIPPER_ATTRIBUTES No \ DO_NOT_AGGREGATE NO \ CLIPPER_ATTR_PREFIX "clipper_" \ OUTPUT INSIDE FEATURE_TYPE CLIPPER_INSIDE \ OUTPUT CLIPPED_INSIDE FEATURE_TYPE CLIPPER_CLIPPED_INSIDE \ OUTPUT EXTRA_CLIPPER FEATURE_TYPE CLIPPER__delete__

@Tcl2("FME_LogMessage fme_warn {Deleting extra clipper feature encountered which entered transformer CLIPPER}") \

OUTPUT NONPOLY_CLIPPER FEATURE_TYPE CLIPPER__delete__ @Tcl2("FME_LogMessage fme_warn {Deleting non-polygonal clipper feature which entered transformer CLIPPER}")

FACTORY_DEF * TeeFactory \ FACTORY_NAME CLIPPER_Nuker \ INPUT FEATURE_TYPE CLIPPER__delete__ \ @Log("Offending feature was:")


Joiner (Relate) ‘Factory’

Relate CACHE_SIZE "JOINER:all_symbology_lookup_table" 5000

Relate TABLE_LOCATION "JOINER:all_symbology_lookup_table" "d:\User Documents\LagenSpatial\Projects\GA - FME Data Migration\Symbology Library (new)\version1.1 20060419\all_symbology_lookup_table.csv"

Relate TABLE_DEF "JOINER:all_symbology_lookup_table" CSV \

FACTORY_DEF * TeeFactory \ FACTORY_NAME JOINER \ INPUT FEATURE_TYPE CLIPPER_INSIDE \ INPUT FEATURE_TYPE CLIPPER_CLIPPED_INSIDE \ OUTPUT FEATURE_TYPE JOINER_JOINED \ @Relate(JOINER,Read) \ @RenameAttributes(_matched_records,matched_records)




Save the pasted output as one or more .fmi files

Use the FMEFactoryPipeline.addFactoriesFromFile() method to include your translation pipeline .fmi files into your PyFMEObjects Script.




WHY???

Reduce possible errors in your script Special Characters

\n – New line\r – Carriage Return\w – White Space\t – Tab !!!

Special characters MEAN something in both Python and FME Mapping Files




Allows you to solve other bugs Logical Programmatic Data Flow

Logical – If…Then…Else Do…Loop Switch…Case





Programmatic – Incorrect Method use String Manipulation Reading Multiple Source Datasets





Data Flow – Combination of Logical and Programmatic Errors Unexpected output




Once you have your Python script reproducing the same output as Workbench, its time to Embed your Factories

…which is also where the fun begins!!




First of all…

Use the Python Lists i.e. stringListVariable = [‘green’, ‘eggs’, ‘and’, ‘ham’]

Cuts out the possibility of naughty special characters sneaking their way into your Factory Definition

i.e. DON’T USE THE “””green eggs and ham“”” strings – it tells Python to use the string as it is, Naughty Special Characters and all


GOOD

fme_session.config(['Relate', 'RELATION_DEF', 'JOINER', '1:0..1+', 'TABLE', 'JOINER:all_symbology_lookup_table', 'UNIQUE(Esri_No,feature_type)', 'JOIN', 'Esri_No', 'TO', 'SYMBOL', 'JOIN', 'feature_type', 'TO', 'fme_type', 'MATCHED_RECORDS_ATTR', 'matched_records', 'TRIM_TRAILING', 'No', 'TRANSFER', 'mapinfo_brush_background', 'TO', 'mapinfo_brush_background', 'TRANSFER', 'mapinfo_brush_foreground', 'TO', 'mapinfo_brush_foreground', 'TRANSFER', 'mapinfo_brush_pattern', 'TO', 'mapinfo_brush_pattern', 'TRANSFER', 'mapinfo_pen_color', 'TO', 'mapinfo_pen_color', 'TRANSFER', 'mapinfo_pen_pattern', 'TO', 'mapinfo_pen_pattern', 'TRANSFER', 'mapinfo_pen_width', 'TO', 'mapinfo_pen_width', 'TRANSFER', 'mapinfo_symbol_angle', 'TO', 'mapinfo_symbol_angle', 'TRANSFER', 'mapinfo_symbol_color', 'TO', 'mapinfo_symbol_color', 'TRANSFER', 'mapinfo_symbol_file_name', 'TO', 'mapinfo_symbol_file_name', 'TRANSFER', 'mapinfo_symbol_font', 'TO', 'mapinfo_symbol_font', 'TRANSFER', 'mapinfo_symbol_shape', 'TO', 'mapinfo_symbol_shape', 'TRANSFER', 'mapinfo_symbol_size', 'TO', 'mapinfo_symbol_size', 'TRANSFER', 'mif_brush_background', 'TO', 'mif_brush_background', 'TRANSFER', 'mif_brush_foreground', 'TO', 'mif_brush_foreground', 'TRANSFER', 'mif_brush_pattern', 'TO', 'mif_brush_pattern', 'TRANSFER', 'mif_pen_color', 'TO', 'mif_pen_color', 'TRANSFER', 'mif_pen_pattern', 'TO', 'mif_pen_pattern', 'TRANSFER', 'mif_pen_width', 'TO', 'mif_pen_width', 'TRANSFER', 'mif_symbol_angle', 'TO', 'mif_symbol_angle', 'TRANSFER', 'mif_symbol_color', 'TO', 'mif_symbol_color', 'TRANSFER', 'mif_symbol_file_name', 'TO', 'mif_symbol_file_name', 'TRANSFER', 'mif_symbol_font', 'TO', 'mif_symbol_font', 'TRANSFER', 'mif_symbol_shape', 'TO', 'mif_symbol_shape', 'TRANSFER', 'mif_symbol_size', 'TO', 'mif_symbol_size',])


BAD

fme_pipeline.addFactoryFromString( """FACTORY_DEF * SDE30QueryFactory \ FACTORY_NAME ARCSDEQUERIER_Querier \ INPUT FEATURE_TYPE __GLOBBED__ \ @SupplyAttributes(__SERVER,""" + server + """,__INSTANCE,port:""" + instance

+""",__DATASET,""" + _dataset + """,__USERID,""" + username + """,__PASSWORD,""" + password + """,__TARGET_TABLE,""" + _owner + "." + featuredataset.upper() + """,__SEARCH_METHOD,SDE_ENVELOPE,__VERSION,SDE.DEFAULT) \

SDE_SERVER_FIELD __SERVER \ SDE_INSTANCE_FIELD __INSTANCE \ SDE_DATASET_FIELD __DATASET \ SDE_USERID_FIELD __USERID \ SDE_PASSWORD_FIELD __PASSWORD \ SDE_VERSION_NAME_FIELD __VERSION \ SDE_TARGET_TABLE_FIELD __TARGET_TABLE \ SDE_SEARCH_METHOD_FIELD __SEARCH_METHOD QUERY_MODE QUERY \ SEARCH_ORDER OPTIMIZE \ COMBINE_ATTRIBUTES MERGE \ GET_SPATIAL_RELATIONS \ OUTPUT RESULT FEATURE_TYPE ARCSDEQUERIER_OUTPUT \

@RemoveAttributes(__SERVER,__INSTANCE,__DATASET,__USERID,__PASSWORD,__TARGET_TABLE,__SEARCH_METHOD,__WHERE_CLAUSE,__VERSION) \

@SupplyAttributes(_table_name,@FeatureType()) \ @Transform(SDE30,FME_GENERIC)""" )




Why Use Python?

Great ‘Whipitupability’

It is an ‘interpreted’ language

Its free and powerful!!! With a great open source community

Supported by ESRI and FME

You can write your own transformers for FME




Example - Sydney




Example - Darwin




Example - Perth




Example – Pt. Lincon




QUESTIONS???

Name: Andrew SmithEmail: [email protected]: +61 2 9545 7740