Mountain Region GIS Advisory Council Meeting September 15, 2006 Long-Term Preservation of Digital Geospatial Data: A Cooperative Project with Library of Congress Steve Morris North Carolina State University Libraries 2 Overview Spatial Data Preservation: Values and Considerations NC Geospatial Data Archiving Project Approaches to Preservation Challenges Workflow NC Spatial Data Infrastructure NC OneMap Regional/Local Partnerships and Data Sharing Coordinated Content Transfer Industry Engagement Historic and Geologic Map Preservation Project 3 Todays geospatial data as tomorrows cultural heritage Future uses of data are difficult to anticipate (as with Sanborn Maps). 4 Temporal Data Supports Decision Making Land use change analysis Real Estate trend analysis Site selection (past uses?) Forecasting Parcel Boundary Changes North Raleigh, NC 5 Time series Ortho imagery Vicinity of Raleigh-Durham International Airport 6 Geospatial Data: Risks Producer focus on current data Future support of data formats in question Shift to web services- and API-based access Inadequate or nonexistent metadata Increasing use of spatial databases for data management Many digital archiving challenges 7 Geospatial data types: Aerial imagery 85+ NC counties with orthophotos 1-5 flights per county gb per flight 8 Geospatial data types: Vector & Tabular Economic, infrastructure, and ethnographic data 9 Geospatial data types: Cartographic Project Files Counterpart to the map is not just the dataset but also models, symbolization, classification, annotation, etc. 10 How would you describe your current geospatial archive? Last weeks set of nightly tape backups Several boxes of CDs and DVDs Bobs hard drive A collection of files in our GIS Folder A stand-alone spatial database The data back-end for our internet mapping application An enterprise GIS 11 NC Geospatial Data Archiving Project (NCGDAP) Partnership between university library (NCSU) and state agency (NCCGIA), with Library of Congress under the National Digital Information Infrastructure and Preservation Program (NDIIPP) One of 8 initial NDIIPP partnerships Focus on state and local geospatial content in North Carolina (state demonstration) Tied to NC OneMap initiative, which provides for seamless access to data, metadata, and inventories Objective: engage existing state/federal geospatial data infrastructures in preservation Serve as catalyst for discussion within industry 12 13 NDIIPP Overview National Digital Information Infrastructure and Preservation Program Congress appropriated $100 million for this effort, which instructs the Library to spend an initial $25 million to develop and execute a congressionally approved strategic plan Eight initial projects, : web pages, cultural heritage, numeric data, video, business records, mixed content, geospatial (2) Developing partnerships and identifying issues Extensive interaction among NDIIPP projects 14 Different Ways to Approach Preservation Technical solutions: How do we archive acquired content over the long term? Tools Hardware Software Cultural/Organizational solutions: How do we make the data more preservableand more prone to be archivedfrom point of production? Collaboration Education Feedback 15 Technical Approaches Receive data as is variety of distribution methods Migration of some at-risk formats Metadata remediation, standardization, and synchronization Distilling complex objects into repository ingest items (not easy) Using DSpace for demonstration purposes In the development: use METS record as dormant item brain within the repository Some unsustainable activities for learning experience 16 Cultural/Organizational Approaches Feedback to metadata outreach program Feedback to coordinating bodies on adherence to content standards Engage existing spatial data infrastructure in archiving and preservation Engage software vendors and standards community Cross-fertilize with other national archiving efforts Current use and data sharing requirements not archiving needs drive improved preservability of content and improvement of metadata 17 Challenge: Vector Data Formats No widely-supported, open vector formats for geospatial data Spatial Data Transfer Standard (SDTS) not widely supported Geography Markup Language (GML) diversity of application schemas and profiles threatens permanent access Spatial Databases The sum is more than the whole of the parts, and the sum is very difficult to preserve Can export individual data layers for curation Some thinking of using the spatial database as the primary archival platform 18 Challenge: Geospatial Web Services How to capture records from decision- making processes? Possible: Atlas collections from automated image capture Web 2.0 impact: Emerging tiling and caching schemes (archive target?) 19 Challenge: Preserving Cartographic Representation 20 General Workflow 1)Receive Data from Agency 2)Copy data from agency source to NCSU workstation 3)Create Dspace collection space for the data 4)Create administrative metadata 5)Process geospatial metadata 6)Scan geospatial formats and migrate to archival format 7)Ingest original and archival data objects, and geospatial administrative metadata to Dspace 21 NCGDAP Leveraging Existing Spatial Data Infrastructure (NC OneMap) NC OneMap: "Historic and temporal data will be maintained and available, RAMONA Metadata outreach and content standards Regional Partnerships WGRT and other Coordination Efforts Data Sharing Agreements Frequent communication and discussion among geospatial data community 22 Challenge: Coordinated Content Transfer How to allow one data snapshot to be accessible by multiple agencies more compelling use cases than preservation can put the data in motion (business continuity, disaster preparedness, etc.) Question: Capture frequency of data snapshot? Survey in progress to identify local government best practices, consumer agencies needs Working Group for Roads and Transportation (WGRT) Stakeholder group working to build data depository for statewide local road data First serious effort to develop a plan for local-to-state data sharing on a regular basis Other Activities? (DHS, State Archives, Census, etc.) 23 Partnership Activity ESRI Discussing software requirements: meetings with development teams April 2005 Open Geospatial Consortium (OGC) Presented to Architecture Working Group Nov National Archives and Records Administration Investigations into GML for archiving; presentation to NARA technology team Dec FGDC Historical Data Working Group Ongoing, general geospatial data preservation issues 24 Partnership Activity EDINA (University of Edinburgh, UK) NCSU is Associate Partner on UK project for geospatial institutional repositories UC Santa Barbara & Stanford University Collaboration with other NDIIPP geospatial project EROS Data Center Planned site visit Project visits to regional GIS groups 25 Preservation of Digital Geologic and Historic Maps Georeferenced over 450 maps scanned by NC Geologic Survey Maps are available for download at15-min topo maps 1:500,000 1:2.5 M 1:31,680 1:430,000 1,200 24,000 26 Questions? Contact: Steve Morris Head, Digital Library Initiatives NCSU Libraries Web site: 27 NC Spatial Data Infrastructure: NCOneMap NC OneMap is a next generation mechanism to coordinate and disseminate geographic information in North Carolina and interact with the NSDI. Objectives: Build a common understanding of North Carolina data resources Enable widespread access and distribution of geospatial data 28 NC OneMap Viewer 29 NC OneMap Objectives (cont.): Develop ongoing data inventory for all geospatial data holdings RAMONA Develop content standards for key data themes NC Geographic Information Coordinating Council (GICC) One of the defined characteristics of NC OneMap is that Historic and temporal data will be maintained and available. 30 Emerging Regional Partnerships Focused on development of shared infrastructure for cultivating access to data Becoming test beds for innovation in the area of data sharing and data management, including archiving 31 Local Govt. Data Sharing Becoming more open, fewer agreements to sign Recent survey: over 20 state and federal agencies use local data Problem of local governments being swamped by requests Many requests are more compelling than archiving Content transfer is non-trivial large dataset sizes, small rural staffs, technical limitations 32 Earlier NCSU Acquisition Efforts NCSU University Extension project Target: County/city data in eastern NC Digital rescue not digital preservation Project learning outcomes Confirmed concerns about long term access Need for efficient inventory/acquisition Wide range in rights/licensing Need to work within statewide infrastructure Acquired experience; unanticipated collaboration 33 Big Geoarchiving Challenges Format migration paths Management of data versions over time Preservation metadata Preserving cartographic representation Keeping content repository-agnostic Preserving geodatabases Harnessing geospatial web services More