Embed Size (px)
DESCRIPTION
In an effort to increase the availability of Topologically Integrated Geographic Encoding and Reference (TIGER) spatial data, the Geography Division of the US Census Bureau has made increasing usage of the ESRI Shapefile format for the exchange of spatial data, both internally and for public products. To accomplish this goal, the basic conversion functionality provided by FME has been utilized in the creation of a number of shapefile products. We will discuss our basic implementation approach, which utilizes manually prepared FME mapping files that feed into a fully automated shapefile creation process and is used for the creation of most of our shapefile products. This process is used to create the Public TIGER/Line Shapefiles, as well as the shapefiles provided to local government partners for updates using the MAF/TIGER Partnership Software. We will also discuss our more recent experience using FME Workbench to design more complicated geo-processing models, which are then saved as FME mapping files and integrated into a similar, automated process. This new process is used to create the shapefiles that serve as the data source for the American Fact Finder web site.
Citation preview
Using FME to Convert TIGER Spatial Data From Oracle Spatial to ESRI Shapefiles
Mei-Leng Freeman, Tim Puzycki, Jay E. Spurlin Spatial Products Software Branch Geography Division, Bureau of the Census
2
Introduction
• Public products – TIGER/Line Shapefiles – Partnership Program Shapefiles
• Internal products – Data Access and Dissemination Staff
(DADS) shapefiles for use in the American Fact Finder websites (internal and external)
– Live Individual Shapefile Request and Delivery System (LISRDS)
– Quality Control (QC) shapefile for various projects
3
Shapefile Creation: Legacy Approach
• Legacy TIGER – Utilized for 1990 and 2000 Decennial Censuses – In-house database format – Binary files – Partitioned by county
• Legacy Shapefile Generation (shpGen) – Java shapefile classes – Perl driver scripts – Directly read from a TIGER partition – Legacy TIGER C API – Directly wrote to a shapefile – ASCII Job configuration files (user defined tags)
4
Modernized Approach
• MAF/TIGER – The Legacy Master Address File (MAF) and
Topologically Integrated Geographic Encoding and Reference (TIGER) datasets were combined.
– Oracle Spatial and Oracle Topology Data Model v10.2
– Seamless, national data set • Current shpGen
– Perl driver scripts – Pre-generated FME mapping files (.FME) – FME Engine handles reading from Oracle Spatial
and writing to ESRI shapefiles
5
Initial Process
• Considerations for automation – Automate everything possible. – Use COTS software, wherever possible. – Ensure consistency of data after conversion. – Provide ease of maintenance. – Make source code reusable. – Fairly stable source data structure. – Fairly stable target shapefile formats.
6
FME Mapping File Usage
• Manually create FME mapping files (.FME) – Previous FME mapping files used as
templates. – A standardized set of macros is used.
• The FME mapping files are released as part of the software installation.
• Automated production control system drives the process and provides any dynamic inputs.
7
Oracle Wallet Support
• Stores database credentials: – Username – Password – Database
• Avoids providing Username and Password to command line interface
8
Example FME Mapping File
9
Example FME Mapping File
Manually Created FME Mapping Files
• TIGER/Line Shapefiles – Produced from “Benchmarks” (snapshots) of
the MAF/TIGER Database. – Public TIGER/Line shapefiles are posted on
the world wide web. • Partnership Shapefiles
– MAF/TIGER Partnership Software (MTPS)
• LISRDS Shapefiles – On-demand access to MAF/TIGER data – Data source may be “live” or a snapshot
10
Current process
• A wider set of shapefile requirements – Mapping performance issues
• Chained line representation of area boundaries • Un-duplication of lines in partitioned layers
– Visual representation issues • Partial features • ‘Balance of’ features
– Connect with Oracle Wallet
11
FME Mapping Files from FME Workbench
• DADS/AFF Shapefiles – Both .FME and .FMW files are archived with
the source code. – The .FME files are released to production,
and used by FME Engine. – A similar procedure (though not yet as
automated, and not making use of the geoprocessing tools) is used by another branch for the generalized and projected shapefiles.
– Samples on the following slides:
12
13
14
15
16
17
18
19
20
Example: DADS/AFF Line Shapefiles
21
State 09
State 05
State 02
State 13
Example: DADS/AFF Line Shapefiles
22
State 09
State 05
State 02
State 13
23
24
Minnesota (State 27)
Iowa (State 19)
25
Workbench File Example: County Subdivisions
26
Example: Connect with Oracle Wallet
Thank You!
• Questions? • For more information:
– Jay E. Spurlin • [email protected]
– Bureau of the Census • 4600 Silver Hill Road, Stop 7400 • Washington, DC 20233
• Products made using FME Engine – TIGER/Line Shapefiles
• http://www.census.gov/geo/www/tiger/index.html – American FactFinder (map base data)
• http://www.factfinder.census.gov/home/saff/main.html?_lang=en
