View
1
Download
0
Category
Preview:
Citation preview
10/12/2010
AmericaView 2010 Fall Technical Meeting
Technical WMS
Dayne Broderson
Maddison WI October 12th, 2010
www.gina.alaska.edu
dayne@alaska.edu
Technical WMS services overview
AmericaView FTM 2010AlaskaView – Dayne Broderson
AlaskaView applications and architecture Data prep WMS overviewMapServerTile services Web mapping GISJAM
AlaskaView WMS
BDL – (best image layer)Blue Marble → Landsat → Highres
USGS ToposNOAA ChartsDEM HillshadeMODIS passes (near-realtime WMS)Many custom off by one layers
10/12/2010
The bdl_high_res layer contains data from a variety of sources and is intended to be viewed at scales from 25m per pixel to 1m per pixel. Images courtesy of Digital Globe, USDA-NRCS, USDA-FS, USGS, BLM, NPS, FAA, Alaska DNR, DMVA, DCED, DOT&PF
http://www.alaskamapped.org/data/wms-wfs-wcs-web-service-feeds
10/12/2010
NOAA ChartsThe Charts service is meant to provide a unified view of the RNC data, without the collar information. The service is designed to only display charts that are meaningful at the requested scale.
http://www.alaskamapped.org/data/gallery
10/12/2010
Shaded Relief NEDThis group has one layer which provides a shaped relief dataset derived from the National Elevation Dataset (NED).
10/12/2010
Realtime MODIS and AVHRRRealtime optical and infrared satellite data captured in Fairbanks and Barrow by UAF-GINA and NOAA-NESDIS FCDAS.
10/12/2010
Not just rasters
Gridded data sets such as DEMs can be served up in a very similar fashion
10/12/2010
2 arc second NEDStandard National Elevation Dataset (NED) for Alaska.
10/12/2010
ASTER GDEM – 1 arc second
10/12/2010
Using it for web applications
Alaska Statewide Digital Mapping Initiative's AlaskaMapped.org
http://browse.alaskamapped.orgNorth Slope Science Initiative's catalog
http://catalog.northslope.orgAlaskaViews user portals for MODIS and firepoint browsing
http://hub.gina.alaska.edu
10/12/2010
show-n-tell
Good place to see it all come together in one place: http://hub.gina.alaska.edusee latest MODIS passview firepoints
10/12/2010
WMS clients
Desktop clients, primarily ArcGIS and QGISServer clients
Alaska DNR's AlaskaMapperAlaska Ocean Observing SystemARMAPNRCS Soil Survey
10/12/2010
Behind the curtain (hardware/OS)
Open standards, open platform, optomized for users and clientsHardware
SuperMicro chassis+motherboardsJBOD RAID + FC RAID
Operating systemCentOS 5x / RHEL (Redhat Enterprise Linux)Trusted repositories (rpmforge+postgresql)
10/12/2010
Behind the curtain (software)
programming language: ruby and javascriptRails and extjs (sencha)
gdal, PROJ4, libgeos, mapserver, ...Postgresql + postgis for database and spatial engine… piles of supporting libraries and software
10/12/2010
Custom software
dans-gdal-toolsCustom tools developed to use in conjunction with GDAL - developed by Dan Stahlke at GINA
swathviewerCustom java application for browsing the data archive of GINA
many custom ruby, c, perl utilities to assist with data ingest and prep.Javascript libraries and REST APIs for simplifying web application development inside and outside of AlaskaView
WMS HA - High availability
Two primary servers, in separate machine rooms attached to their own independent mirror of the data.
High Availability - HA provided by using:two HA IPs shared between the serversDNS round robin between HA IPsHA IP failover
Allows us to bring down a WMS, do massive data update, and switch over to updated dataset w/o service interruption. secret sauce: Linux HA (heartbeat)
Tile Servers
Similar to WMS HA setup changed up to optimized for tile clients. A HA setup pointed at two load balancing proxy server
Apache → proxy → one of many instances of shiv
secret sauce: apache (and it's built in proxy capability)
Preparing Data
Some things to evaluate/figure out before shoving your data at a WMS
map product no data valuesmask/outline of valid dataprojection/datum of sourceresolution of source optimized projection/file-format/pyramiding of data
no-data | valid data
Knowing which pixels in a file are valid data pixels is very important for creating a layered WMS. What is the outline of your dataset? (mask)What are the correct 'nodata' value? (or values!)
For just looking at a single image no big deal - layering in a WMS having no-data pixels be opaque will block pixels from other layers you want - or having data pixels labeled as no-data will cause voids where lower layers can peek through (ugly artifacts) secret sauce: gdalinfo
gdalinfo - nodata
gdalinfo 55012391009071952112J0_gm.tif Driver: GTiff/GeoTIFF
Files: 55012391009071952112J0_gm.tifSize is 7450, 7390
----------(lots more info here)--------Band 1 Block=7450x1 Type=Byte, ColorInterp=Red
NoData Value=0
Band 2 Block=7450x1 Type=Byte, ColorInterp=Green
NoData Value=0
Band 3 Block=7450x1 Type=Byte, ColorInterp=Blue
NoData Value=0
Band 4 Block=7450x1 Type=Byte, ColorInterp=AlphNoData Value=0
gdalinfo - projection
PROJCS["NAD83 / Alaska zone 3", GEOGCS["NAD83", DATUM["North_American_Datum_1983", SPHEROID["GRS 1980",6378137,298.2572221010002, AUTHORITY["EPSG","7019"]], AUTHORITY["EPSG","6269"]], PRIMEM["Greenwich",0], UNIT["degree",0.0174532925199433], AUTHORITY["EPSG","4269"]], PROJECTION["Transverse_Mercator"], PARAMETER["latitude_of_origin",54], PARAMETER["central_meridian",-146], PARAMETER["scale_factor",0.9999], PARAMETER["false_easting",1640416.666666667], PARAMETER["false_northing",0], UNIT["US survey foot",0.3048006096012192, AUTHORITY["EPSG","9003"]], AUTHORITY["EPSG","26933"]]
if only...
The gdalinfo trick only works if the stars align:image format selected allows for that sort of embedded metadatathe people generating the data used correct options allowing for the fields to be populated
Many data formats you have to look in an auxiliary metadata file. With the same issues as above.
Many older data products suffer from complete lack of required metadata -- Sherlock Holmes hat and pipe.
updating state: prepped -> create_images/data/scratch/tracker-prep/54542271009182120192J0Warping scene01/imagery.tif -> sdmi/54542271009182120192J0_aa.tif
running: >>> gdalwarp -dstnodata '0 0 0' -t_srs epsg:102006 -rb -co TILED=YES -co COMPRESS=LZW scene01/imagery.tif sdmi/54542271009182120192J0_aa.tif
Generating RGB sdmi/54542271009182120192J0_aa.tif -> sdmi/54542271009182120192J0_aa_rgb.tif
running: >>> gdal_translate -a_nodata '0 0 0' -b 1 -b 2 -b 3 -co TILED=YES -co COMPRESS=LZW sdmi/54542271009182120192J0_aa.tif sdmi/54542271009182120192J0_aa_rgb.tif
Generating pyramids for sdmi/54542271009182120192J0_aa_rgb.tif
running: >>> gdaladdo -r average sdmi/54542271009182120192J0_aa_rgb.tif 2 4 8 16 32 64 128 256 512 1024 2048 4096
Fetching metadataCreating tagger.ymlupdating state: create_tagger_yml -> prepping
What did we just see?
gdalwarp: Raw SPOT image rubber-sheeted info an Alaskan Albers projection in a 4 band geotif
gdal_translate: Converting the 4 band geotif to a RGB product in Alaskan Albers gdaladdo: Create internal pyramids/overviews for the RGB geotif (make it speedy for ArcGIS users and WMS).
running: >>> gdalwarp -dstnodata '0 0 0' -t_srs '+proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0 +k=1.0 +units=m +wktext +no_defs' -rb -co COMPRESS=DEFLATE scene01/imagery.tif /home/ingest/data/gm/54542271009182120192J0_gm.tif
generating cornersrunning: >>> gdal_list_corners /home/ingest/data/gm/54542271009182120192J0_gm.tif > 54542271009182120192J0_gm.yml Generating RGB /home/ingest/data/gmrunning: >>> gdal_translate -a_nodata '0 0 0' -b 1 -b 2 -b 3 /home/ingest/data/gm/54542271009182120192J0_gm.tif /home/ingest/data/gm/54542271009182120192J0_gm_rgb.tif
creating preview imagerunning: >>> tifftopnm /home/ingest/data/gm/54542271009182120192J0_gm_rgb.tif | pamscale -xyfit 1300 1300 | pnmtopng -transparent=black > /home/ingest/data/preview/54542271009182120192J0_gm_rgb.png
What was that?
That was doing a similar set of steps only this time the goal was a Google Mercator geotif.
Also saw a handy tool called gdal_list_corners being used. Like gdalinfo only output format is YAML making it easily parse-able by scripts.
Lastly we saw the conversion of the GM RGB to a resized 1300x1300 PNG using tifftopnm | pamscale | pnmtopng Why? http://photos.brute.gina.alaska.edu
-Data-
Know your data and it's attributes
Have that information in an easy to use format (human and scripts). -- YAML much easier than XML
Get confused and bang around on the command line but keep track of the commands and script them up for large job.
WMS features
Sam hopefully did this already
on the fly re-projection (server side)customized layers/mosaicsprovide access to vast archive w/o downloadupdate 'best available' and be able to update what that meansprovides a great back-end to tile based work
WMS limitations
Bandwidth and latency between server and clientsTimeouts by clients ESRI 'features'
no user control over image formatinternal timeouts w/o errorsuncontrolled caching by clientbad defaults for behavior re-projection and re-sampling algorithms
Above add up to limited large printing capability (plotting)Client side proxies/network setup can cause a nasty headache when debugging
WMS Server Options
Open SourceGeoServer - http://geoserver.org
java based - MapServer
ProprietaryESRI ArcGIS Server, ArcIMSOracle, LizardTech, ....GeoWebPublisher, ObjectFX, ...
MapServer
http://mapserver.org/
University of Minnesota "The MapServer Project Steering Committee (PSC), sanctioned by OSGeo, manages and administers the project which is maintained, improved, and supported by developers from around the world. MapServer is not a full-featured GIS system, nor does it aspire to be."
secret sauce: yum install mapserver (not actually but you could to get started)
MapServer Architecure
Ingredients for a WMS:
Pile of raster mapsPile of vectors (not covered here)Apache serverMapServerA CGI/URL Map file (organization/configuration for a WMS)
above shamelessly stolen/borrowed from mapserver's introduction documentation
What did we see there?
Data sources:raster and vector data on local machineWFS, WCS, WMS - remote servers
Mapfile configuration
list of what data sources to include Map server CGI (MapScript application ignored)
Apache server
Output - the WMS/WCS/WFS
Dirty hands - data store
bishop@salt:~/mapserv/data$ ls
AERIAL.UAF.MAY2007.UAF_AREA_2_12AERIAL.UAF.MAY2007.UAF_AREA_2_13bmSPOT.DNR.INTERIOR.TANANA_VALLEY
Mapfile - real world examples
BDL example
pile of examples (restricted URL)
mapfile - phase 1
MAP NAME "AView Example" # SIZE 600 400 # pixel width and height of the map EXTENT -180 -90 180 90 # geographic IMAGECOLOR 128 128 255 # default image background colorEND
map file - data layer
LAYER NAME "Blue_Marble" STATUS on DATA "/home/bishop/mapserv/data/bm/blue_marble_bright.tif" TYPE RASTER PROJECTION "init=epsg:102006" END END
the cgi
The following are some example CGI's used by AlaskaView:
simple real world example
the BDL cgi (more magic)
my aview example
#!/bin/sh MAPFILE="/home/bishop/mapserv/aview.map" MAPSERV="/usr/lib/cgi-bin/mapserv" if [ "${REQUEST_METHOD}" = "GET" ]; then if [ -z "${QUERY_STRING}" ]; then QUERY_STRING="SERVICE=WMS&map=${MAPFILE}" else QUERY_STRING="SERVICE=WMS&map=${MAPFILE}& STYLES=&${QUERY_STRING}" fi exec ${MAPSERV} else echo "Sorry, I only understand GET requests." fi exit 1
aview cgi
REQUEST=GetCapabilities
curl http://localhost/cgi-bin/aview?REQUEST=GetCapabilities
--Provide clients an XML description of what is offered.
File formats supportedLayers available...
curl http://wms.alaskamapped.org/extras?REQUEST=GetCapabilities
msProcessProjection(): Projection library error. no options found in 'init' file root@salt:/usr/share/proj# grep 102006 esri<102006> +proj=aea +lat_1=55 +lat_2=65 +lat_0=50 +lon_0=-154 +x_0=0 +y_0=0 +ellps=GRS80 +datum=NAD83 +units=m no_defs <>
gah - that needs to be in /usr/share/proj/epsg grep 102006 esri >> epsg
Optimizing WMS
Watch how your clients are using the WMS
Avoid re-projection on the flyUse TIFFS with overviews/pyramidsminimize load-up time of the cgi/mapserver environmentuse 'ab' to benchmark your changesenable cachingMany different map files (lean/optimized) are better than one massive map file
optimizing rasters faq @ mapserver.org
Expert words
General tips for performance all cases:First and foremost is hardware. An extra GB of RAM will give your map performance increases beyond anything you’re likely to achieve by tweaking your data. With the price of RAM these days, it’s cheap and easy to speed up every map with one inexpensive upgrade.Use the scientific method. Change one thing at a time, and see what effect it had. Try disabling all layers and enabling them one at a time until you discover which layer is being problematic.Use shp2img program to time your results. This runs from the command line and draws an image of your entire map. Since it’s run from the command line, it is immune to net lag and will give more consistent measurements that your web browser.
http://mapserver.org/faq.html
Tile Generators
Tile generators take tile requests and map them to WMS requests
Maps a request for a tile: http://www.foo.com/tilesets/x/y/z
To a WMS request:http://www.foo.com/wms?layers=mylayer&styles=&service=WMS&format=image...
Why? Allows you to use a WMS to provide fast and efficient tiles to web mapping frameworks
Tile Generators
A Tile Generator shouldDo cachingProvide a way to "seed" the cacheGenerate several tiles from a single WMS request
ExtrasKML and ESRI Rest Style output
Tile Generator Options
Tile Cache http://tilecache.org/
ShivESRI Arc ServerProbably lots of other good options..
Shiv - overview
Written in Ruby Built by Jay Cable for AlaskaView/GINAProven to be very flexible and stable
shiv instances with 100+ day uptimeability to add/remove/revamp what tile layers are available w/o restarting of services can handle all major tile schemes
Shiv
featuresorganized tile cache structure
Tiles reside on disk in a tree structure pre-generates tiles optimizes requests to WMS
requests a large image that it chops upFlexible tile support (power of 2)
Such as Google Maps Generates the KML for Google Earth Speaks ESRI style REST
ExperimentalNot a recommended direction to go in
An Example
Shiv receives a request for the Google Maps style tile at 0,0,1 (x,yz) Having no pre-generated tile to serve up Shiv requests the following image from the WMS
An Example
Taking that large image Shiv generates these tiles..
An Example
then returns the tile, 0,0,1:
Nice - but how to use it?
GINA.mapTypes = { SDMI_BDL: new google.maps.ImageMapType({ name: 'GINA/SDMI Best Data Layer', alt: 'GINA/SDMI Best Data Layer', minZoom: 1, maxZoom: 21, tileSize: new google.maps.Size(256, 256), isPng: true, getTileUrl: function(a, z) { var tiles = 1 << z, X = (a.x % tiles); if(X < 0) { X += tiles; } return 'http://swmha.gina.alaska.edu/tilesrv/bdl/tile/' + X + '/' + a.y + '/' + z + '.png'; } }), ...
Tile URL
That was kinda hard to read - broken apart:
http://swmha.gina.alaska.edu /tilesrv/bdl/tile/' + X + '/' + a.y + '/' + z + '.png';
/tilesrv/bdl/tile/0/0/1.png
/tilesrv/bdl/tile/1/3/4.png
/tilesrv/bdl/tile/341/1161/12.png
...
Time to switch to the 'make a map' portion of this show.
Prepare yourself a text editor and have firefox ready to go to some URLs
http://gisjam.gina.alaska.edu/
gisjam slides
Recommended