Python-based Solutions to Maintain Enterprise Data Currency at the Bureau of Land ManagementBy Adam Ridley, GIS Specialist

Cottonwood Field Office – Idaho BLM

2

Overview

► BLM Mission and Structure

► Idaho BLM GIS Architecture and Directory/File System

► Data Management Challenges– 4 Components

► Background and Issues

► Coding Approach

► Timeline

► Results

► Questions

3BLM Mission & Structure

Washington Office

Idaho State Office

Cottonwood Field Office

11 Additional Field Offices

12 Other State/Region

Offices

Many Local Field Offices

Bureau-wide

State Office

Field Office

Mission:To sustain the health, diversity, and productivity of America’s public lands for the use and enjoyment of present and future generations.

Photo Credit: Craig Johnson, Fishery Biologist

4Idaho BLM

► 12 million acres in Idaho

► 1 National Conservation Area

► 1 National Monument

► 16 Wild and Scenic Rivers covering 318 miles

► 8 Wilderness Areas covering 541,812 acres  

► 4 National Historic Trails covering 452 miles  

► 1 National Scenic Trail covering 13 miles

5

Idaho BLM GIS Architecture

► Esri-based enterprise GIS using SDE

► Data pertains to Cultural Resources, Forestry, Range, Recreation, Wildlife, etc.

► Largely rural state, limited telecom infrastructure

► History of bandwidth issues and network instability

► Local file servers at each Field Office; local ArcGIS Desktop installations; local data

► Limited Web GIS development

Photo Credit: Ryan Turner, Park Ranger

6

Directory/File System

Server RootGIS Root

Project Example

7

Data Management Challenges

► Challenge 1: Layer files reference incorrect fields or have broken links

► Challenge 2: Local corporate datasets become decadent due to problems with replication

► Challenge 3: Metadata is not typically included in the one-way replication process

► Challenge 4: Changes to corporate data or archiving projects breaks links in map documents

8

► Layer files function by storing a snapshot of a layer’s properties

► Idaho BLM maintains a library of layer on each local server

► Roads, Fires, Boundaries, Ownership, etc.

► Layer files link to both replicated and local reference data

► Changes to reference data can break layer files

Layer File ValidationChallenge 1

9

Replica CurrencyChallenge 2

► Replication – Esri process to manage changes across an enterprise GIS.

► One-way replication

► State Office to Field Offices

► Runs Nightly

► Uncertainty results from:

► Not all datasets are set for regular replication.

► Nightly replication can fail for a variety of reasons

► Replication failures not reported to Field Office GIS staff

State Office server

Field Office server

One -wayReplication

10

Metadata CurrencyChallenge 3

► Similar issues to Replica Currency

► Metadata changes not captured through replication process

► Updates to metadata occur when replica is rebuilt or when pushed manually.

► Either scenario infrequent at best

11

MXD ValidationChallenge 4

► MXDs are an Esri file format – maintain symbolized data and map layouts

► MXDs store file paths one of two ways:

► An absolute path (“R:\loc\GIS\PROJECTS\Heckman CE\Heckman Inclusions Oct_2014.shp”)

► A path relative to the location of the MXD (“.\HeckmanInclusionsOct_2014.shp”)

► Links break if:

► MXD or referenced data is moved

► Referenced data is renamed

How do we maintain viability of MXDs over long term and support archiving?

► Final Data favors absolute paths; project data favors relative paths

12

Why Code? Why Python?

► Ability to schedule the script for off-hours execution

► Reduces likelihood for schema locks

► Less network competition

► Consistency in task execution

► Personnel time and associated costs with manual process

► Department of Interior (DOI) and BLM IT policies require extensive vetting process for applications

13

Coding Approach - General

► Familiar access and feel through ArcToolbox

► 4 scripts, 4 tool interfaces

► May develop a stand-alone GUI using wxPython

► Options to Analyze & Report or Analyze & Fix for each script

14

Coding Approach – Layer File ValidationChallenge 1

Solution: Validate layer files and fix or remove broken links

Check the data source and symbology value fields arcpy.mapping.ListBrokenDataSources (Tilton 2011) and arcpy.Describe (Watkins 2014)

Find correct dataset to re-link• d

atasetName property of Layer objects

15

Coding Approach – Replica CurrencyChallenge 2

Solution: Test our local geodatabases against the State Office data stack for currency and update as needed

Index both the replica and parent geodatabases• a

rcpy.da.Walk function

Test analogous feature classes’ similarity• M

odified dates

• arcpy.Describe objects

• Feature Compare (Data Management > Data Comparison > Feature Compare)

Update if indicated

16

Coding Approach – Metadata CurrencyChallenge 3

Solution: Check for metadata updates on the State Office data stack and replace local metadata as needed

Update metadata• (

Conversion Tools > Metadata > Import Metadata)

17

Coding Approach – MXD ValidationChallenge 4

Solution: Test MXDs for broken links and repair or replace data as needed

Locate and index MXDs• a

rcpy.da.Walk

Collect layers with broken links• a

rcpy.mapping.ListBrokenDataSources

Re-llink to appropriate data source• R

elatively simple for Final Data; may use look-up table

• Difficult for project data, particularly if renamed

18

Timeframe

Jan-Feb •2 weeks of development time for each of the 4 components, totaling 8 weeks of development time

Mar •Additional 4 weeks for documentation, testing/debugging, GUI development, etc.

Apr •Presentation to Idaho Leadership Committee

19

Results

► Similarity in server structures allow for state-wide application

► Confidence in our local data’s accuracy and currency

► Less time spent fixing data and projects

► Reliability of data stores for end users

20

References

Tilton, Thad. Charming the Snake: An Introduction to Python and the ArcPy site-package. ArcUser Vol. 14 ,No. 1 (Winter 2011), p. 53-55.

Watkins, David. Using Python to Gather Information about Data in SDE. 2014 Esri User Conference Proceedings.

21

Questions?Photo Credit: Craig Johnson, Fishery Biologist