Upload
geosolutions
View
3.509
Download
0
Tags:
Embed Size (px)
DESCRIPTION
The purpose of this presentation is to discuss the developments during last years in raster data support in GeoTools and GeoServer, and also to introduce and discuss future development directions.
Citation preview
Raster data in GeoServer and GeoTools: Achievements, issues and future developments
Ing. Simone Giannecchini, GeoSolutions Ing. Daniele Romagnoli
FOSS4G 2013, Nottingham 21th September 2013
Outline
Who we are
The building blocks
Recent Achievements
What’s cooking
FOSS4G 2013, Nottingham 21th September 2013
GeoSolutions
Founded in Italy in late 2006
Expertise
• Image Processing, GeoSpatial Data Fusion
• Java, Java Enterprise, C++, Python
• JPEG2000, JPIP, Advanced 2D visualization
Supporting/Developing FOSS4G projects
MapStore, GeoServer
GeoBatch, GeoNetwork
Clients
Public Agencies
Private Companies
http://www.geo-solutions.it
FOSS4G 2013, Nottingham 21th September 2013
The Stack we’ll talk about
FOSS4G 2013, Nottingham 21th September 2013
JAI
Java Advanced Imaging Library
SUN then Oracle Image Processing library
Huge number of basic building operations
Tile-based, deferred execution model
Integrate Tile Caching (In Memory)
Easily Extensible
Natively Accelerated Code available for many operations
No source code for native ops
Development more or less stopped!
FOSS4G 2013, Nottingham 21th September 2013
ImageIO
SUN then Oracle Image I/O library
Partly ships with JDK partly as extension
Tile-based, deferred & immediate execution model
Easily Extensible
Natively Accelerated Code available for many formats
No source code for native (de)coders
Development more or less stopped!
FOSS4G 2013, Nottingham 21th September 2013
JAITools
High performance raster processing
Extends, complements, replaces Oracle Java Advanced
Imaging (JAI) Library
Open Source, BSD license
Low Level Operations
e.g. RangeLookup
Higher Level Operations
Vectorize, Contour
Works mostly in pure raster space (Image Processing)
Stable release is 1.3.0
FOSS4G 2013, Nottingham 21th September 2013
ImageIO-Ext
High Performance raster I/O
Extends, complements, replaces Oracle JAI ImageIO
Library
Open Source, LGPL and BSD licenses
Plugins:
BigTiff*, MatFile 5, TurboJPEG
JP2000 (Kakadu)
GDAL integration via ImageIO-ext
JPEG2000, MrSID, ECW
BigTIFF, ERDAS Image, HDF4
Complete list here
FOSS4G 2013, Nottingham 21th September 2013
GeoTools
The Open Source Java GIS Toolkit
LGPL
Management of Geospatial data
CRUD Operations
Raster and vector
Rendering
GDAL Support
Based on best-of-breed Open Source projects
Java Advanced Imaging (JAI), ImageIO
Java Topology Suite (JTS)
FOSS4G 2013, Nottingham
21th September 2013
GeoServer
GeoSpatial enterprise gateway
Java Enterprise
Management of raster and vector data
Standards compliant
OGC WCS 1.0, 1.1.1 (RI), 2.0.1
OGC WFS 1.0, 1.1 (RI), 2.0
OGC WMS 1.1.1, 1.3.0
OGC WPS 1.0.0
Google Earth/Maps support
KML, GeoSearch, etc..
FOSS4G 2013, Nottingham 21th September 2013
All together now…
GeoTools
JTS ImageIO
GDAL
WCS
JAI
WMS WFS WPS REST Google
Vector Raster
GWC
Style
Kakadu JAITools
FOSS4G 2013, Nottingham 21th September 2013
Recent Achievements
FOSS4G 2013, Nottingham 21th September 2013
ImageIO-Ext
LibJPEG-Turbo derivative library of libjpeg based on SIMD
instructions (MMX, SSE2, NEON)
accelerate baseline JPEG compression and decompression on x86, x86-64, and ARM systems
available on many platforms
Open Source, LGPL
ImageIO-Ext-TurboJPEG A JPEG ImageIO plugin built on top of the JAVA
wrapper for LibJPEG-Turbo (TurboJPEG)
20%-30% SpeedUp* on JPEG encoding
10%-20% SpeedUp* on JPEG decoding
More info here
Integrates with TIFF reader and writer
FOSS4G 2013, Nottingham 21th September 2013
ImageIO-Ext
NITF encoding Support (ImageIO-Ext-NITF)
Based on NITRO
Support for Tagged Record Extensions (TREs)
Support ImageSegments JPEG2000 compression through Kakadu ImageIO-Ext plugin
(Numerically Lossless/VisuallyLossless/Lossy)
NPJE, EPJE BIIF (Basic Image Interchange Format) profiles
J2KLRA TRE support
Support for multiple TextSegments
Support for multiple ImageSegments
Support for ShapeFile Data Extension Segment (CSSHPA)
FOSS4G 2013, Nottingham 21th September 2013
GeoTools
Band Combine + Symbology
Raster to point vector
On-the-fly Contouring
On-the-fly Polygonalization
Improved internal JAI Concurrency
Introduced and Improved JAI Tile Recycling
Reduced depth of JAI Processing Chains
More aggressive factory caching
Added a few specific high perf JAI operations
e.g. crop
Created Specific high perf ROI class
Improving Tile Caching
Work ongoing
Transparent File vs Memory Caching when writing
raster
FOSS4G 2013, Nottingham 21th September 2013
GeoTools
Optimized Rendering/Processing for raster data
Optimized Crop Chains Coalescing multiple Crops into a single one
Optimize Affine Transformation Chains Coalescing multiple Affine Transformations into a single one
FOSS4G 2013, Nottingham 21th September 2013
NEAREST
GeoTools
FOSS4G 2013, Nottingham 21th September 2013
BICUBIC
GeoTools
ImageMosaic
Custom Dimensions More than simple time/elevation support
E.g.: wavelength, cloudLevel, …
Dimensions with ranges: (startTime, endTime), (minElevation,maxElevation), …
Dynamic read parameters
Filtering
Supports for OGC Filters through the “filter” param E.g.: Return only ImageMosaic granules where
attribute Country = Italy.
filter=“Country=‘Italy’ ”
FOSS4G 2013, Nottingham 21th September 2013
ImageMosaic
Custom/Additional dimensions
FOSS4G 2013, Nottingham 21th September 2013
GeoTools
ImageMosaic
Ordering
SORT_BY clause support on DBMS based mosaics
sorting=“time A, elevation D”
Stacking VS Merging
Merging: different granules merged together in the final image as default
Stacking: different granules stacked on different bands
E.g. MeteoSat data: 12 channels (1 granule for each Channel) 1 = VIS0.6, 2 = VIS0.8, …, 11 = IR13.4, 12 = HRV.
Want return a RGB image made of channels 1,3,7 Stack them
FOSS4G 2013, Nottingham 21th September 2013
GeoTools
New API (unsupported/coverage-experiment)
Driver: Get a connection to a coverage store.
CoverageAccess: a physical storage of coverage data. Get access to a source of coverages.
CoverageSource: provides data access to a coverage.
Updated GridCoverage core API
Old GridCoverageReader:
inefficient and thread unsafe way to access multiple coverages
read GridCoverages from the input stream in a sequential order
New GridCoverage2DReader:
Coverages and properties may be accessed by coverage name:
get Envelope(name), get GridRange(name), get CRS(name), read(name), get dynamicParameters (name), …
dynamicParameters support (for custom dimensions)
FOSS4G 2013, Nottingham 21th September 2013
GeoTools
NetCDF Support
Based on Unidata NetCDF JAVA library
Implements the new coverage-experiment API
Exposes current interfaces (GridCoverage2DReader,…) too for GeoServer compatibility before changes
1 coverage for each Dataset’s variable (multiple coverages).
Internal data exposed as set of 2D slices
Custom dimensions support
Ancillary XML file:
filter on NetCDF variables/coverages
define low level index schema (for dimensions)
FOSS4G 2013, Nottingham 21th September 2013
ImageMosaic NetCDF integration
Allow the ImageMosaic to handle NetCDF files
Expose NetCDF internal structure (times, elevations)
Make ImageMosaic handle slices of the NetCDF file as granules
Mosaic
NetCDF 1 NetCDF 2 NetCDF … N
…
GeoTools
FOSS4G 2013, Nottingham 21th September 2013
GeoTools
NetCDF Polyphemus Sample Dataset
FOSS4G 2013, Nottingham 20th September 2013
GeoTools
FOSS4G 2013, Nottingham 21th September 2013
Multiple Coverages per Mosaic
GeoTools
StructuredGridCoverage2DReader exposes the underlying granule structure and dimensions
descriptors of a specific coverage
getGranules (return a GranuleSource)
getDimensionDescriptors
allows to create/remove coverages and add new sources
createCoverage
removeCoverage
Harvest
GetGranules
FOSS4G 2013, Nottingham 21th September 2013
ImageMosaic
FOSS4G 2013, Nottingham 21th September 2013
Granule Index CRUD Operations via REST
CREATE curl -u admin:Geos -XPUT -H "Content-type:application/zip" --data-binary @http://localhost:8080/geoserver/rest/workspaces/geosolutions/coveragestores/t
emperature/file.imagemosaic
READ index schema
curl -v -u admin:Geos -XGET "http://localhost:8080/geoserver/rest/workspaces/geosolutions/coveragestores/polyphemus/coverages/NO2/index.xml"
READ WFS like with CQL filtering and paging
curl -v -u admin:Geos -XGET "http://localhost:8080/geoserver/rest/workspaces/geosolutions/coveragestores/polyphemus/coverages/NO2/index/granules.xml?limit=1&filter=time='2013-03-
03T00:00:00Z'"
ImageMosaic
FOSS4G 2013, Nottingham 21th September 2013
Granule Index CRUD Operations via REST
UPDATE curl -v -u admin:Geos -XPOST -H "Content-type: text/plain" -d "/polyphemus_20130303.nc" "http://localhost:8080/geoserver/rest/workspaces/geosolutions/coveragestores/polyphemus/external.imagemosaic"
DELETE WFS like with CQL filtering and paging or by ID
curl -v -u admin:geoserver -XDELETE "http://localhost:8080/geoserver/rest/workspaces/topp/coveragestores/polyphemus-v1/coverages/NO2/index/granules.xml?filter=location='polyphemus_20130301.nc'"
curl -v -u admin:geoserver -XGET "http://localhost:8080/geoserver/rest/workspaces/topp/coveragestores/polyphemus-v1/coverages/NO2/index/granules/NO2.2689.xml"
GeoTools
GranuleSource/GranuleStore GranuleSource:
ReadOnly access to Granules (with Query support)
getGranules(Query) SimpleFeatureCollection
getSchema SimpleFeatureType
getCount(Query), getBounds(Query)
E.g.: get all granules having acquisitionYear > 2012 and Country=Italy
GranuleStore:
add/modify/delete capabilities using Transactions
addGranules (SimpleFeatureCollection)
removeGranules (Filter)
updateGranules
E.g.: remove all granules having acquisitionYear < 1990
FOSS4G 2013, Nottingham 21th September 2013
GeoServer
LibJPEG-Turbo Output format
Encoding WMS output JPEG images through LibJPEG Turbo library
Based on the TurboJpeg ImageIO-Ext plugin built on top of the JAVA interface for LibJPEG-Turbo (TurboJPEG)
Provided as a GeoServer extension
Up to 80% SpeedUp measured on JPEG encoding
Can be disabled at startup:
-Ddisable.turbojpeg=true
More Info here
FOSS4G 2013, Nottingham 21th September 2013
GeoServer
More Color Quantizers
Speed loss is around 10%
FOSS4G 2013, Nottingham 21th September 2013
OCTTREE
MEDIANCUT
GeoServer
WMS Custom Dimensions Work on GeoTools raster Additional Domains
NetCDF
ImageMosaic
Turn additional querying dimensions into WCS/WMS Dimensions
ImageMosaic Domain Category
WMS 1.1.1 DIM_CATEGORY
WMS 1.3.0 DIM_CATEGORY
Perfect for large multidimensional mosaics!
E.g. meteo (forecast time, elevation, band, base time, model name)
FOSS4G 2013, Nottingham 21th September 2013
GeoServer
FOSS4G 2013, Nottingham 21th September 2013
WMS TIME, ELEVATION & More http://localhost:8080/geoserver/geosolutions/wms?...&time=2013-03-01T00:00:00.000Z&elevation=35.0&DIM_FILEDATE=2013-03-01T00:00:00.000Z&DIM_UPDATED=2013-04-08T08:18:41.597Z
GeoServer
Extending LayerGroup concept
Support same style on both raster and vector
data
Support custom dimensions
Alter map on the fly to support band combination
Wizard to configure EO layer groups
WMS-EO GSIP 84
FOSS4G 2013, Nottingham 21th September 2013
GeoServer
WMS-EO Quirks
Root layer must respond with a specific layer rather than the composition of the children
Kind of a default visualization for a certain Dataset
E.g. when I send a GetMap for the root layer I could get back the browse image default layer
Same styles for both Raster as well as Vector data
E.g. yellow can be use for both flags as well as outlines
Peculiar behavior for Band layer ( raw data )
E.g. multiple bands at different wavelengths
Can request either 1 (grayscale image) or 3 (RGB image)
Different combinations are prohibited
Peculiar Behavior for GetFeatureInfo
FOSS4G 2013, Nottingham 21th September 2013
GeoServer
WCS 2.0 The basics
Core service KVP binding XML binding
GetCoverage extensions CRS Scaling Interpolation Range subsetting GeoTiff GML NetCDF
Output format extensions GeoTIFF GML Grid NetCDF
Ensure CITE tests compliancy
FOSS4G 2013, Nottingham 21th September 2013
GeoServer
WCS 2.0
Processing Chain
Crop
RangeSubset
Scale & Interpolate
Reproject & Interpolate
Encode
Core
Extension
Extension
Extension
Extension
FOSS4G 2013, Nottingham 21th September 2013
GeoServer
NetCDF Output Format GetCoverage request parameter: &Format=NetCDF
Based on Unidata NetCDF JAVA library
Create a NetCDF output file on top of a GranuleStack:
A GridCoverage2D List:
a GridCoverage2D for each combination of requested Dimension values
GridCoverages should come from a StructuredGridCoverage2DReader
Supports WCS 2.0.1
Trimming: specify values range for a dimension
Slicing: specify single value for a dimension
Supports timeRanges/numberRanges on output
FOSS4G 2013, Nottingham 21th September 2013
GeoServer
NetCDF Output
http://localhost:8080/geoserver/wcs?request=GetCoverage&service=WCS&version=2.0.1&coverageId=geosolutions__NO2&Format=NetCDF&subset=http://www.opengis.net/def/axis/OGC/0/Long(5,20)&subset=http://www.opengis.net/def/axis/OGC/0/Lat(40,50)&subset=http://www.opengis.net/def/axis/OGC/0/elevation(300,1250)&subset=http://www.opengis.net/def/axis/OGC/0/time("2013-03-01T10:00:00.000Z","2013-03-01T22:00:00.000Z")
FOSS4G 2013, Nottingham 21th September 2013
GeoServer
NetCDF Output
FOSS4G 2013, Nottingham 21th September 2013
GeoServer
WCS-EO Build on top of a working WCS 2.0 with full extensions
WCS 2.0
CRS extension
Range subsetting extension
Interpolation extension
Scaling extension
GeoTiff extension
NetCDF extension
Add support for the WCS-EO extras
Listing coverage datasets in the capabilities documents (based on image mosaic contents, which will have to be marked as “exposed” so that we show their inner structure for EO)
Support describe coverage dataset
Support returning results for an entire dataset in GetCoverage
FOSS4G 2013, Nottingham 21th September 2013
What’s cooking Plus what’s on our wish list
FOSS4G 2013, Nottingham 21th September 2013
JAI-Ext (temporary name)
Oracle JAI Drop-In Replacement
Open Source (License TBD)
High Performance
Pure Java
Support for No-Data
Support for Processing ROI
Support for Band Masks
GPU Acceleration
OpenCL VS JCuda VS Aparapi
FOSS4G 2013, Nottingham 21th September 2013
JAI-Ext (temporary name)
Status Mosaic
Scale
Affine
Translate
Lookup
BandSelect
BandMerge
Warp
ColorQuantize
Format
FOSS4G 2013, Nottingham
21th September 2013
Rescale
Histogram
Extrema
….
GeoTools
Improved Processing & Rendering
Raster Algebra embedded in Styles
Processing on-the-fly with a Rendering Transformation
Embed into RasterSymbolizer (ChannelSelection)
Improved Contrast Stretch
Clip to Min-Max, Gaussian, etc…
Suport for Date-line crossing raster
Support for raster in Polar projections
Better No-Data Management
Support for VirtualRaster a-la GDAL
FOSS4G 2013, Nottingham 21th September 2013
GeoTools
Mosaic Masking
FOSS4G 2013, Nottingham 21th September 2013
GeoServer
High Performance/Pure Java PNG output format
No more separate Native ImageIO Install
More Control on bugs/enhancements
Status
First prototype working
Same performance as Native Imageio
Much Faster then JDK Encoder (check this)
Code here, GPL|
Discussion ongoing with the PNGJ Open Source project for joining efforts
More info here
FOSS4G 2013, Nottingham 21th September 2013
GeoServer
FOSS4G 2013, Nottingham 21th September 2013
OSM Generated Image
GeoServer
FOSS4G 2013, Nottingham 21th September 2013
Syntethic Image
GeoServer
Raster Algebra as WPS Gets an OGC Filter containing logical operations between
raster layers and produces a binary (0/1) layer
FOSS4G 2013, Nottingham 21th September 2013
GeoServer
Raster Algebra as WPS JIFFLE
FOSS4G 2013, Nottingham 21th September 2013
The End
Thanks for not sleeping (loudly)!
FOSS4G 2013, Nottingham 21th September 2013