Embed Size (px)
Citation preview
Back to FME School3-Day webinar series
Course ScheduleTuesday:
Your Data and FME
Wednesday: Automating Everything
Thursday:Expanding Frontiers
FME Workbench Performance Tips & Tricks
A Database for Every Occasion
Working with Attributes in FME
Bulk data processing
Ultimate Real-Time: Monitor Anything, Update Anything
FME in the Enterprise
Remote sensing providers
Web and business data into gold
New + Upcoming in FME Cloud
40-minute talks
Not your average school...
FME Workbench Performance Tips
& Tricks
Double, DoubleLess Toil, Less TroubleGIS professionals are finding themselves tasked with doing double the work in half the time!
Luckily, FME is designed to magically ease that burden.
Problem: TheDeadline Dragon
Only FME stands between you and being roasted by your clients …
Solution: The Performance Pixies
… FME 2017 boasts the best pixie powers yet
PerformanceGoals
✓ Create an FME solution faster than ever.
✓ Execute it faster than ever.
Performance Highlights for FME 2017
“Turn to Page 394”
A Quick List● Remodeled CSV Reading● Improved Attribute Management● Better List Handling● Enhanced Polygon Dissolving● Faster FeatureReader
Workflows● Empowered Reading
of Data From Web Sources
In Action
“It’s still magic, even if you know how it’s done”
Goal: Add geographical information to the list of FME World Tour stops.
Steps1. Read a Geographical Names CSV Dataset
2. Filter the Geographical Names Data for a List of Cities Only
3. Read the Current List of FME World Tour Events
4. Merge the FMEWT Event with its Country Code
5. Match the FMEWT Event to its Host City
6. Ensure the Correct Location is chosen for each Event
Here’s the workspace to solve that problem.
2016
• Read Data: 1 minute 35 seconds• Read/Filter: 1 minute 36 seconds
2017
• Read Data: 8.3 seconds• Read/Filter: 14.1 seconds
Processing Times
SummaryPerformance in Creation• Attributes to Read• List Element Selection• HTML Table Reader• Select Source from Web• Modeless Parameter Dialogs• Quick Add Readers/Writers• User Parameter Copy/Paste• Open Containing Folder
Performance in Execution
• Attributes to Read• List Element Selection• Feature Table Formats• Feature Table Transformers• Faster Polygon Dissolving• Faster FeatureReader
Questions?
Learn More:knowledge.safe.com
blog.safe.com
A Database for Every Occasion
Types of Databases
. . . For Every Occasion
Relational Sports league
NoSQL Epic party
File-based Camping
Relational Databases
ID Name Position
1 Harry Potter Seeker
2 Ginny Weasley Chaser
3 Draco Malfoy Seeker
ID Name Colors
1 Gryffindor Red and gold
2 Slytherin Green and silver
TeamID PlayerID
1 1
1 2
2 3
Player
Team
Team_Players
Relational Databases
✔ Static schema ✔ Server Infrastucture ✔ Accuracy ✔ Complex SQL querying ✔ Control security for each table
NoSQL Databases
{
ID:1,
Title:"BilboBaggins’EleventyFirstBirthdayParty",
Location:Shire,
FireworksDisplay:{
Name:"GandalftheWizard",
Time:"halfpastsix",
}
}
AWS DynamoDB
NoSQL Databases
✔ Agile schema ✔ High data volumes ✔ Fast transactions ✔ Scalable ✔ Access via APIs
File-Based Databases
File-Based Databases
✔ Simple ✔ Shareable ✔ Tiny ✔ Extensible
FME 2017 Updates
FME 2017 – New Formats New Formats: • MBTiles (R/W) • OGC GeoPackage Raster (R/W) • MapInfo Extended TAB (R) • IBM dashDB (R/W)
DatabaseJoiner
Attributes to Read
Demo: Bald Eagle Survey A team of wildlife surveyors needs to assess eagle habitat.
Which type of database to use?
OGC GeoPackage • Free and open source • Standards-based • Platform-independent • Portable • Compact
Supports raster and vector all in a SQLite Database! http://www.geopackage.org/
Completing the Survey
1. 1) Build the Database
2. 2) Collect Data
3. 3) Combine Changes
Step 1: Build Database using Existing Data
Step 2: Collect Data
Go Mobile!
Place your screenshot here
http://ngageoint.github.io/geopackage-mapcache-android/
Final Step: Merge all Updates into Master
FME Tips: 1. Detect updates 2. fme_db_operation 3. Unique ID field
Story: Field Data Collection and Maintenance
Step 1: Field Data Collection
Step 2: Transform AGOL to Excel & File Geodatabase
Esri Survey123
Initial Import + Transform: AGOL > File GDB
Iterative Import + Transform: File GDB > File GDB + Excel
Step 3: Cyclical Quality Control
✅
Result: Master data repository + Excel / Power BI
Summary
• Choose the right database for the occasion – Pros and cons to relational,
NoSQL, and file-based databases
• Update master database using UpdateDetector, fme_db_operation, and a unique ID
Questions?
Learn more:
knowledge.safe.com Tutorial: Updating Databases
NoSQL blogs + webinar
safe.com/nosql
Time Machines and Attribute
AlchemyMark StoakesSafe Software Inc.
Date & Time Attribute Manipulation
1. Dates and times
RCMP E DivisionHeidi Lee | Robert Shultz
Goal: Load GPS records into ArcGIS.
Problems:➔ Inconsistent date formats➔ Time zones➔ Daylight saving
Dates and times are complicated.
Formatting?
● YYYYMMDDHHMMSS.SS (FME)
● <d v="2016-12-13T00:00:00"/> (Excel)
● YYMMDD, HHMMSS, UTC
● Jun 2016
● ‘on Saturday, Jan 9th 2016, 01:00 am’ & ‘+0530’
● 2016-12-07 12:20:07.785403-05
● 20160313020000.000 (March 13 - Daylight Saving)
● YYYY-MM-DD hh:mm:ss[.nnnnnnn] (SQL Server ‘datetime2’ value)
Calculations?
● Date2-Date1 = How many days?
Improved FMEDate Functions
Read
TransformParseProcessFormat
Write
Read
TransformParseProcessFormat
Write
Over to FME
Time Zones
FME 2017UTC Offsete.g. -08:00
FME 2018IANA Time Zone
e.g. America/Vancouver
Attribute Alchemy
Last Year
Attribute Management
Quality Control
Southern CompanyJeff DeWitt
HOK Inc.David Baldacchino
Goals:
➔ Test for patterns in attribute values
➔ Extract substrings from attribute values
➔ Validate strings
2. Finding patternsNGI BelgiumJan Beyen
RCMP E Div.Heidi Lee
Southern CompanyProblem: Attribute value cleanup
- MONTANA * or Sales/Other (1)
HOK Inc.Problem: Extract Sheet numbers from file names
- MyProject - Sheet - A512 - PARTITION TYPES & … - G001 - GENERAL NOTES, ABBREVIATIONS, SYMBOLS, ...
NGIProblem: Validate address strings
- Rue Achille Masset 52A
Sheet number extraction
MyProject - Sheet - A512 - PARTITION TYPES & … G001 - GENERAL NOTES, ABBREVIATIONS, SYMBOLS, ...
^[\S\s]*?[-]?[ ]?([A-Z0-9]{1,5})[ ]+[-]+[\S\s]*$
Address validation
Rue Achille Masset 52A
^((([a-zA-Z]+) )+)([0-9]+)([a-zA-Z]*)$
Wave the wand of regex
Regular ExpressionEditor...
...making regexeasier sinceFME 2016!
Alternatively: String functions
Regex vs. String Functions Example
Code ABD3705337067
Regular Expression: ([A-Z]{3})([0-9]+)
String Functions:
Attribute String Function
alpha @Left(@Value(Code),3)
beta @Substring(@Value(Code),3,-1)
StringPairReplacer
Replace multiple values from a list, for example:
Niccolo' Giosue' would become Niccolò Giosuè
More String Replacement options...
TRC Inc.Peter Veenstra
Goals:
➔ Compare current and previous Excel rows.
➔ Sum attribute values with the previous row.
3. Time-travelling attributes.
FPInnovationsMatt Kurowski
Past, Present & Future Attributes
feature[-1].measure measure
feature[+1].measure
Where to find it:AttributeManagerAttributeCreator
Workspace: Pipeline Weld Reconciliation
Referencing the last feature’s valuesFor the attributes stationing, measure & odometer, calculate the cumulative values from the previous features
Attribute Aggregation ChallengeRule: If T is 3 or less, aggregate this row with the row above.
Attribute Aggregation Challenge
Summary1. DateTime transformers and Text Editor
functions help with:○ Date/time formatting○ Calculations○ Time zones
2. Regex and string functions help with patterns.
3. Work with current and previous attribute values in the AttributeManager.
That wraps up today’s classes...
Course ScheduleTuesday:
Your Data and FME
Wednesday: Automating Everything
Thursday:Expanding Frontiers
FME Workbench Performance Tips & Tricks
A Database for Every Occasion
Working with Attributes in FME
Bulk data processing
Ultimate Real-Time: Monitor Anything, Update Anything
FME in the Enterprise
Remote sensing providers
Web and business data into gold
New + Upcoming in FME Cloud
40-minute talks
Questions?
Tutorial: fme.ly/datetime
AttributeManagerdocumentation
