ArcGIS Data Modelsdownloads.esri.com/support/ProjectCenter/ArcGIS_Data... · 2004. 10. 2. · Raster Data Model • Data Properties: – Source Data: • 440 GB total raster input

ArcGIS Data Models:Raster Data Models

ArcGIS Data Models:Raster Data Models

Jason Willison, Simon Woo, Qian Liu(Team Raster, ESRI Software Products)

Jason Willison, Simon Woo, Qian Liu(Team Raster, ESRI Software Products)

Overview of Session• Raster Data Model Context• Example Raster Data Models• Important Raster GDB Considerations• Rasters in ArcGIS 9.0

Questions / Answers / Comments

Sample Database and Design DocumentsFor more information, go to

http://support.esri.com/datamodels

What is a Data Model?

“A practical template for implementing GIS solutions”

• Built on consensus within a discipline– Both providers and users

• Goal is consistency across organizations• Great starting points• Designed to adhere to emerging standards

Why Data Models?• Data models provide…

– “Quick Start" solutions– Best practices

• Optimized performance– Improves data sharing – More successful GIS implementations

• The ArcGIS geodatabase architecture allows…– Ready-to-use frameworks– Centralized location– Intelligence/Rules

Raster Data Model Context

• Raster Data Models• Essential parts of other data models

• Just another layer in your data model stacks• Base-mapping and/or Analysis needs

• ESRI’s Geodatabase Raster Solution• Enterprise or personal geodatabase

• We will focus on enterprise (ArcSDE)• Easy to build specifically to meet your needs

Raster Data Model Context

Demonstration

Where do I start?

• Determine…– The purpose of my raster data– Your application’s scale of use– The properties of the raster data

• All this contributes to your database design– Use these guidelines to optimize your solution

Types of Geodatabase Rasters

• Raster Dataset– Storage of a single raster dataset– Mosaic many inputs to one continuous raster dataset

• Raster Catalog– Container of independent raster datasets– Organized as one entity in the database

• Raster Attributes of a Feature Class– Feature class attribute– Stored in the geodatabase

Raster Datasets

• Single geodatabase raster dataset created from multiple data sources– Single row table in the geodatabase – Overlapping pixels have options (overwrite, ignore, blend, min, max)– Use for DEMs, most base-mapping, and where fast viewing at any

scale is important

3712a.tif 3712b.tif

3712c.tif 3712d.tif

UC2004.IMAGE_3712

Input files Geodatabase Raster Dataset

Raster Catalogs

• Collection of Raster Datasets stored in one table– Designed for…

• Overlapping data (time series, stereo-pair photos, scanned maps)• Situations where storing each dataset independently is important• When you want to organize similar data

– Use entire catalog, or a subset

IMAGE NAME DATE1 WC_4724 05/26/982 WC_4725 07/05/013 WC_4726 10/04/014 WC_4727 02/26/02

ArcSDEArcSDERasterRaster

columncolumn

UC2004.AERIAL_CATALOG

Raster Attributes

• Geodatabase raster data is associated with geodatabase features

Personal Geodatabases

• Raster Data in the personal geodatabase– Pixels are not stored inside Access

• Raster Data is stored as referenced files– Managed

• Raster Data is copied as IMGs to a special folder**Note: Raster Datasets are always managed

– Unmanaged• Raster Data is simply referenced by a pathname**Note: Raster Catalogs and attributes can be managed or

unmanaged

Demonstration

Enterprise GDB, Personal GDB, or Raster Files

• Enterprise GDB – Our best multi-user solution – Large data volumes (terabytes)– Multi-platform / multi-database

• Personal GDB – Good for prototyping and small implementations– Limited number of connections– Small quantity of data (gigabytes)– Windows only

Storage / Hardware

Two local servers (here at UC2004)

• 100 megabit network

• DELL PC• Dual 933 MHz processors• 1GB of RAM• 1TB normal disk storage

• IBM PC • Dual 3 GHz processors• 4GB of RAM• connected to an INLINE storage device

– 2 disk arrays, 1.6TB each

Raster Data Model Case Studies

• Ortho-rectified Imagery• Global Raster • Raster Elevation• Scanned Map• Time Series• Raster Attributes

Ortho-rectified ImageryTopography Data Model• Cartography• Administrative areas• Parcels

– Uses, rights/interests, and ownership

• Legal description• Corners and boundaries• Survey control• Vectors (roads, rail, buildings)

• Ortho-rectified Imagery– Very accurate imagery

Topography Data Model• Cartography• Administrative areas• Parcels


• Legal description• Corners and boundaries• Survey control• Vectors (roads, rail, buildings)

• Ortho-rectified Imagery– Very accurate imagery

Ortho-rectified Imagery

Hydrography Data Model• Streams• Hydrographic points• Drainage areas • Vector water features• Channels• Surface Terrain/Elevation• Hydro response

• Ortho-rectified Imagery

Hydrography Data Model• Streams• Hydrographic points• Drainage areas • Vector water features• Channels• Surface Terrain/Elevation• Hydro response

• Ortho-rectified Imagery

Ortho-rectified Imagery Data Model

(Example Implementation)

• Texas Natural Resources Information System (TNRIS)– Responsible for managing data for all of Texas– 1 meter Color-Infrared – Digital Ortho-rectified Quarter Quads (DOQQs)– Used a Raster Catalog – ~ 440 GB of raw data

• 3000 of 16000 provided to ESRI

Demonstration

TNRIS Ortho-imagery Raster Data Model

• Purpose:– Base-Mapping

• Update schedule:– Irregularly scheduled updates– Has portions of the state irregularly updated each year

by the counties of Texas(** Not part of the decision process anymore in 9.0 **)


• Scale of Use:

(** assumes user will not use lossy compression **)

(Table courtesy of EMERGE)

Seems a bit too conservative for monitors, OK for printing

Orthorectified Imagery Scale of Use Considering Lossy Compression

6 15 25 50 75 941 : 1000

1 : 5000

1 : 12000

1 : 24000

1 : 63360

1 : 150000

1 : 250000

more loss… JPEG Compression …less loss

M A P

S C A L E


• Database Design:– Enterprise Geodatabase

• Size of the data • Personal would not have scaled enough

– Raster Catalog • To preserve their base-mapping unit• To preserve overlapping corners

– Lossy compression (JPEG)• Base-mapping (not for analysis)

– Cubic Convolution pyramid resampling• Continuous data


• Data Properties:– Source Data:

• 440 GB total raster input data• 3000 TIFF files (150 MB each)

– Enterprise Geodatabase Data:• 135 GB in the Geodatabase• Raster Catalog

– ArcGIS Layer:• Use scale dependency to view • Primarily 10 to 20 rasters at a time• Up to 1 : 3780 (1:1 screen to pixel in the monitor)

– Printing would be different, inline with industry recommendations

L.A. County Ortho-imagery Data Model

• Very similar to TNRIS– Base-mapping

• Lossy compression (JPEG)

• Differences…– 0.6 meter resolution – Natural color– Bilinear Interpolation pyramids– Required fast display at any scale

• Raster Dataset

Demonstration


• Ortho-rectified Imagery• Global Raster• Raster Elevation• Scanned Map• Time Series• Raster Attributes

Global Raster(Example Implementation)

• EarthSat’s SNC of the world– Contains all land masses– 15m resolution– Approximately 4.5 TB of data

• ~3600 TIFF files (all >1GB) • Used 9 independent Raster Datasets• Mosaicked a portion of their files into each

Demonstration

Global Raster

• Very similar to TNRIS– Base-mapping

• Lossy compression (JPEG)– 1 meter resolution (Natural Color)– Cubic Convolution pyramids

• Required fast display at any scale

• Also required fast loading period– 9 Raster Datasets– 9 clients loading data into 1 server– < 1 week to load into the geodatabase



Raster Elevation Data

Hydrography Data Model• Streams• Hydrographic points• Drainage areas • Vector water features• Channels

• Surface Terrain/Elevation

• Hydro response• Orthorectified Imagery

Hydrography Data Model• Streams• Hydrographic points• Drainage areas • Vector water features• Channels

• Surface Terrain/Elevation

• Hydro response• Orthorectified Imagery

Raster Elevation Data Model (Example Implementation)

• Southern California Association of Governments (SCAG)– 5 meter raster elevation dataset– L.A. County

• provided by Emerge/Intermap

Demo

Raster Elevation Data Model

• Purpose: – Derive additional datasets (analysis)

• Update Schedule:– Very infrequent updates(** Not part of the decision process anymore in 9.0**)


(Table courtesy of USGS)

Scales of Use:Because these are interpolated values, these Scales set by the USGS are valid thresholds.

Based on this information, we can infer that 5m data is valid for mapping up to about 1:4000.


• Database Design:– Raster Dataset

• Mosaic together• Overlap is the same surface, so it does not need to be

preserved– Lossless compression (LZ77)

• Data will be derived (analysis)– Cubic Convolution Pyramid resampling

• Continuous data

Raster Elevation Data Model• Data Properties:

– Source Data• 2.25 GB, 2 GRIDs• 5 meter Intermap DSM (surface)• RADAR product

– Geodatabase Data• One large raster of 3 GB• Lossless LZ77

– ArcGIS Layer• Use at any scale (up to 1:4000)• Derive other layers as needed (slope, aspect)



Scanned Maps

Topography Data Model• Cartography• Administrative areas• Parcels


• Legal description• Corners and boundaries• Survey control• Vectors

– roads, rail, buildings• Ortho-rectified Imagery

• Scanned topographic maps

Scanned Map (Example Implementation)

• National Geographic Society TOPO! data– Scanned USGS Topographic Maps– 7.5 degree extents (1 : 24000 series)– Southern California

Demonstration

Scanned Map Raster Data Model

• Purpose: – Base-mapping

• Update Schedule:– Infrequent updates(** Not part of the decision process anymore in 9.0**)


• Scales of Use:– Map Scale– 1:24,000 = 7.5 min (4’ per pixel @ 500dpi)

– 1:63,360 = 15 min (10.5’ per pixel @ 500dpi)

– 1:100,000 = 30 x 60 min (16.7’ per pixel @ 500dpi)

– 1:250,000 = 30 x 60 min (41.7’ per pixel @ 500dpi)

• Scanned map: useable at the scale it was made – Have an inherent scale of use – DO NOT get increasingly better at higher scan rates


• Database Design:– Raster Dataset

• Stored as RGB – Inputs had unmatching colormaps – Converted Single-band with colormap to multi-band RGB(** Not necessary anymore in 9.0**)

– Lossy compression • JPEG 50• Not used for analysis

– Bilinear Interpolation pyramid resampling• Produced the best looking output for display


• Data Properties:– Source Data

• 1000’s TPQs converted to 1000’s TIFFs• 1 : 24000 Topographic Maps (scanned by NGS/Topo!)• ~ 3 GB of data

– Geodatabase Data• Raster Dataset • ~ 10 GB (grows to this size because of RGB storage)

– ArcGIS Layer• Use at 1:24000



Raster Time Series

Raster Time Series (Example Implementation)

• Hurricane Data– Hurricane Mitch (late Oct – early Nov, 1998)– Shows time slices of hurricane path

• USGS data– AVHRR NDVI data for continent of Africa– Every 10 days averaged over the last 21 years

Demonstration

Time Series Raster Data Model

• Purpose: – Viewing of same extent at different times

• Update Schedule:– Always being updated – Just add new data to a raster catalog(** Not part of the decision process anymore in 9.0**)


• Scale of Use: – Determined by the data– In our cases…

• Regional / Continental / Global extents and applications


• Database Design:– Raster Catalog

• multiple layers, same extent– Lossless compression

• LZ77• Used for visualization, but could be used for analysis

– Resampling• Cubic Convolution


• Data Properties:– Source Data

• 30 input TIFFs for Mitch• >700 BILs for Africa

– Geodatabase Data• Raster Catalog

– Needs to be a Raster Catalog– Same extent, with multiple captures in time

– ArcGIS Layer• Use 9.0 built-in time-series renderer

– 8.3 custom code on developer samples • Regional / Continental / Global scales



Rasters as Attributes ofFeature Classes

• Geodatabase features can store Raster Datasets

• One or more raster datasets associated with each feature

• Raster data is stored with the vector data– One central location

Demonstration

Overall Decisions / Considerations• Before loading, consider these issues…

– Compression• Purpose of Raster data

– Base map v. analysis– Highest quality v. Disk space

• Will you serve/sell the raster data?• Input raster format

– Heterogeneous v. Homogeneous• Cell size, theme, type, colormap, etc

– Pyramid resampling• Discrete v. continuous

– Preserve overlapping data?

Conclusions

• Raster Data Models are…– simple on their own– part of other data models– easy to implement and use

– Follow these “best-practices” guidelines• Prototype your solution first

Evaluations

Please fill them out……your comments help ESRI become

better at meeting your needs each year

THANK-YOU !

End of PresentationThank-you for attending!

Open to Questions