Upload
safe-software
View
590
Download
2
Embed Size (px)
Citation preview
Automating
Everything with FME
Agenda
• What is Automation?
• Data Loading
Scenarios– On Demand
– Scheduled
– Real Time
• Working with Big Data
• Making Data Available
The Journey of Data Superheroes
Episode #1:
A New Movement
I have systems that aren't connected and data lying
around. I need to start moving data.
The Journey of Data Superheroes
Episode #1:
A New Movement
I have systems that aren't connected and data lying
around. I need to start moving data.
Episode #2:
Transformation Menace
These systems are too different and I can’t easily move
data between them.
The Journey of Data Superheroes
Episode #1:
A New Movement
I have systems that aren't connected and data lying
around. I need to start moving data.
Episode #2:
Transformation Menace
These systems are too different and I can’t easily move
data between them.
Episode #3:
Validation Strikes Back
The data in these systems sucks. I need to filter out the
bad, fix it, and make sure only good stuff gets moved.
The Journey of Data Superheroes
Episode #1:
A New Movement
I have systems that aren't connected and data lying
around. I need to start moving data.
Episode #2:
Transformation Menace
These systems are too different and I can’t easily move
data between them.
Episode #3:
Validation Strikes Back
The data in these systems sucks. I need to filter out the
bad, fix it, and make sure only good stuff gets moved.
Episode #4:
Automation Awakens
I want data to be moved automatically. It should
happen each night, as soon as data is available, or on-
demand if someone needs it right then.
FME Server is automation.
What is Automation?
Running an
FME WorkspaceDelivering DataTriggering an event
Your Top 3 Automations*
1. Loading Data
2. Processing Large Data
Volumes
3. Making Data Available
* based on 5 years of survey data from FME Server Customers
Loading Data• On Demand
• Scheduled
• In Real Time
Example: Loading On Demand
Example: Scheduled Replication
“FME Server is the caretaker that lets us pay attention to other things.”
Example: Loading in Real Time
Data Loading Tips
✓ Validate data before it’s loaded and
immediately report the quality of a user’s
submission.
✓ For duplicate systems, nightly replication
avoids disruptions during the day.
✓ Keeping systems connected in real time is
better for insight/transparency.
Processing Data Volumes.
Large
Colossal
Big
Gigantic
Huge
Enormous
Gargantuan
Big Data Symptoms
❏ Difficulty figuring out how to handle all your data.
❏ So much data but not enough ability to process it.
❏ Regular use of the term terabyte.
❏ Work with imagery, LiDAR, networks,
live streams.
❏ Performing tasks like modelling, spatial
relationships, change detection, large-
scale map production.
Example: Big Networks (10,000 route queries!)
Example: Big Maps
“The volume of data required to be processed needed an automation workspace run on FME Server.”
Big Data Tips
✓ Look for ways to improve performance in
your FME workspaces.
✓ Use a dedicated FME Server machine.
✓ Use 64-bit FME for full access to memory
resources.
✓ Use multiple FME Engines for concurrent
processing.
✓ Use multiple machines, if you got ‘em.
✓ Estimate the total time and frequency to
decide if automation is worth it.
Example: Big Maps in the Cloud
Tip: Put Big Data in the Cloud.
✓ Scalable ● On-demand processing power.
✓ Reliable● 99.99%+ uptime.
✓ High performance● On-demand upgrade of processing power.
✓ Cost effective● Pay only for what you need.
22 Minutes to Destination
Example: Real-time Networks
Brake pressure OK
Fuel Level at 60%
JMS
“Fast Data” Tips
✓ Use multiple FME Engines for
scalability.
✓ Use message queue software
(JMS, SQS).
✓ Keep FME Engines always
running.
Making Data Available.
Consider: Do you need a
map focus, or a
metadata focus?
Consider: Can you offer
more than just the data?
● FME Server was created for automation.
● Automation can mean event triggers, running a
workflow on a schedule, or making data available on
demand.
● Leverage the full processing power of FME Desktop, FME
Server, and FME Cloud to get the most out of your
automation scenarios.
More info about all of the stories seen here:
● On-Demand Data Loading example (Swedish EPA): http://www.safe.com/fmeuc/automated-quality-controls-with-fme/● Scheduled Replication example (San Jose): http://www.safe.com/customers/san-jose-water-company/● Data Loading in Real Time example (San Jose): http://www.safe.com/presentation/oracle-database-triggers-and-fme-server-for-
task-tracking/● Big Networks example (London Fire Brigade): http://www.1spatial.com/wp-
content/uploads/2015/11/London_Fire_Case_Study_FINAL.pdf● Big Maps example: http://www.safe.com/presentation/national-map-production-with-mapnik-rasterizer/ ;
http://www.imgs.ie/ordance-survey-ireland● Big maps in the cloud example (Weather): http://www.slideshare.net/SafeSoftware/fme-around-the-world-46744668/53 ;
http://blog.safe.com/2015/02/event-driven-architecture-aws-lambda-s3/● “Fast data” railway example: http://blog.safe.com/2014/12/125000-mph-railways/● On-demand analytics: https://prezi.com/33sm_x4x6wvz/fme-on-demand-and-automated-spatial-reporting/● Open data, map focus: http://mslapps.mt.gov/geographic_information/data/databundler/ui_cdp● Open data, metadata focus:
http://findit.york.ca/opendata/portal/#/search?disp=f3566bc6&view=card&sortdir=desc&fq=fss_Categories:Spatial%5C%20Data
Thank you!
Questions?