WMS Performance Shootout 2009

OSGeo Presentation

LabelGeoServerMapServer FCGI

1014.318.6

2014.919.4

4014.719.1

???Pagina ??? (???)10/22/2009, 13:51:24Pagina / RasterMaps/secThroughput - 1Throughput - 10Throughput - 20Throughput - 40

bigtiff cgi 564.813.31515.2

bigtiff fcgi 548.22425.224.3

bigtiff fcgi 5611.126.928.127.3

ecw cgi 56412.612.812.6

ecw fcgi 543.39.69.48.9

ecw fcgi 567.42019.819.1

tiled tiff cgi 562.71111.411.3

tiled tiff fcgi 546.616.116.717

tiled tiff fcgi 566.514.916.416.9

Polygon DrawingMaps/secThroughput - 1Throughput - 10Throughput - 20Throughput - 40

shape6.515.11515.5

Shape 54 fcgis12.727.227.928.2

shape 56 fcgi1225.826.226.8

pg5.914.71515.2

pg fcgi10.123.22424.3

oracle3.511.912.512.8

oracle fcgi9.121.121.621.8

oracle ogr0.71.71.71.7

sde1.19.410.110

sde fcgi8.822.422.923.3

Point DrawingMaps/secThroughput - 1Throughput - 10Throughput - 20Throughput - 40

shape6.614.815.215.6

shape 54 fcgi12.927.928.328.7

shape 56 fcgi12.226.526.627.1

pg6.115.115.315.5

pg fcgi10.72424.324.8

oracle3.29.910.711.3

oracle fcgi9.823.223.323.7

sde1.19.710.49.9

sde fcgi9.623.323.824.2

Andrea Aime (GeoServer)
Jeff McKenna (MapServer)WMS Performance Shootout

Executive summary

Compare the performance of WMS serversGeoServer

MapServer

In a number of different workloads: Vector: plain polygon, point with graphical icons, road network with cased lines, road network with labels

Raster: global imagery

Against different data backends: Vector: shapefiles, PostGIS, Oracle spatial, SDE over Oracle spatial

Raster: ECW and GeoTiff mosaics

Benchmarking History

3rd FOSS4G benchmarking exercise. Past exercises included:FOSS4G 2007: Refractions Research run and published the first comparison with the help of GeoServer and MapServer developers. Focus on big shapefiles, postgis, minimal styling

FOSS4G 2008: OpenGeo run and published the second comparison with some review from the MapServer developers. Focus on simple thematic mapping, raster data access, WFS and tile caching

Friendly competition: goal is to improve all software

Past MapServer improvements

improvements in large shapefile indexing

raster read optimization (single pass for multiple bands)

enhancing polygon labelling placement

EPSG codes caching

PostGIS improvements

Oracle improvements

Past GeoServer improvements

Overall rendering pipeline improvements

GML writing speed-ups

Raster data access improvements

PostGIS data store optimizations

Rules of engagement

Each server is tested in its latest version

Each server performs exactly the same workloadSame set of WMS requests

Same data backends

Same image output format

All modifications made to improve performance are to be included in a future release of the software

Data used cannot be modified for display, other than indexing

All testing to be done on the same benchmarking machines

Hardware Configuration

JMeterGeoServerMapServerShapefilesECWGeoTIFFOraclePostGISSDE

Database

Software Specs

MapServer:5.6.0-beta3, GDAL SVN

Apache 2.2

FastCGI - 8 static processes

GeoServer:2.0-RC2, Java 6 update18, GDAL 1.4.5

Tomcat 6.0.20 - 8 threads

Database:Oracle 11.1.0.7

SDE 9.3.1

PostGIS 1.4.0

Hardware specs

Bench (2003):Dell PowerEdge 17501 Xeon Nocona 3Ghz, 1 GB RAM, OS: RedHat Enterprise 5.3

WMS (2004):Dell PowerwEdge 2850

4 Xeon Nocona 3.4Ghz, 8 GB RAM

6 73 GB, 15K RPM hard drives, OS: RedHat Enterprise 5.3

Database (2007):Dell Optiplex 755 tower

1 x Intel Core2 Duo CPU E6750 @ 2.66GHz / Dual Core, 4Gb RAM

100Gb SATA 3.0Gb/s 7200 rpm HD, OS: RedHat Enterprise 5.3

Hot vs Cold Benchmarks

Hot benchmarkThe file system caches are fully loaded

The software has had the occasion to perform whatever pre-processing is necessary (e.g., open connections to the data source)

Cold benchmarkThe system has been just started

There is no existing cache

Both are unrealistic, production is usually a mix of both

Hot benchmarks are more practical to run

Methodology

Each test run performs requests with 1, 10, 20 and 40 parallel clients (for a total of 1500 requests)

Each test uses a random set of requests stored in a CSV file: no two requests in the same run are equal, but all servers perform the same workload

For each request the random factors are:The image size (between 640x480 and 1024x768)

The geographic envelope (extent)

Each test is run three times in a row, the results of the third run are used for the comparison: this benchmark assumes full file system caches (hot benchmark)

The other GIS server is shut down while the tests are run

Datasets Used

Polygon layer: areawater_merge: the TIGER 2008 set of polygons describing water surfaces for the state of Texas. 380000 polygons, 195MB shapefile, EPSG:4269

Point layer: gnis_names09: all location and points of interest names for the state of Texas in the GNIS database. 103000 points, 2.8 MB shapefile, EPSG:4269

Line layer, edges_merge: all line elements (rivers and roads) from the TIGER 2008 dataset for the state of Texas. over 5M lines, 1.2GB shapefile, 1.4GB dbf, EPSG:4269

Raster layer: Bluemarble TNG, 86400 x 43200 pixels, 7 overview layers (TIF, ECW, tiled 512&8, GeoRaster)

Equivalent traffic table

The presentation charts results as throughput in requests per second.

This table translates hits per second at sustained peak load in terms of requests per hour and per day

T1: plain polygon rendering

TIGER 2008 areawater layer, merged over Texas

Simply fill the polygon in blue, no outline

Image scales roughly between 1:10.000 and 1:100.000

T1: Shapefiles

T1: PostGIS

T1: Oracle

T1: SDE

T1: observations

GeoServer is between 25% and 50% faster reading shapefiles out of the box, MapServer with FastCGI is slightly faster

PostGIS performs roughly 5% better than Oracle on higher loads

SDE seems to introduce no overhead compared to direct Oracle access for GeoServer, and a slight overhead for MapServer

GeoServer and MapServer have exactly the same performance when the backend is PostGIS and Oracle

T2: point rendering

GNIS names dataset over Texas

Each point is rendered using a little GIF/PNG graphics

T2: shapefiles

T2: PostGIS

T2: Oracle

T2: SDE

T3: roads map

TIGER 2008 edges, just roads (rivers filtered out), three different line styles based on the road class

T3: shapefiles

T3: PostGIS

T3: Oracle

T3: SDE

T4: labelled roads

Same as T3, but higher scale (between 1:10k and 1:3k) and with labels following lines

T4: shapefiles

T4: PostGIS

T4: Oracle

T4: SDE

T5: raster

Bluemarble TNG, as single ECW file, 512 mosaic TIFF, and BigTIFF (MapServer) vs 8 tiles mosaic TIFF (GeoServer)

ECW file is 260MB

TIFFs are uncompressed, tiled, with overviews. Around 16GB for each mosaic

T5: ECW

T5: 512 tiles mosaic

T5: BigTiff vs 8 tiles tiff

GeoServer cannot do BigTiff, so a 8 tiles mosaic was used (almost 2G B per tile)

FOSS4G 2009 improvements

Several code enhancements were made during this year's exercise:MapServer Fixed problem with labels following features

Fixed problem with 5.6.0-beta for SDE layers

Found problem with 5.6.0-beta for Oracle fastcgi and memory leak

GeoServerFixed problem with Oracle connection pooling and index scanning

Fixed problem with ECW crashes under high load

Hopeful Changes for Next Shootout

Involvement of more WMS servers

Put the servers under a faster internet connection for quicker data transfers and tests setups

Modify the number of parallel client requests performed so that the 2-4-8 parallel clients case gets exposed

Phases/cycles for benchmarking to allow for teams to interpret results and make software enhancements

Reprojection in both vector and raster data

Audience suggestions?

More Information

Scripts, configuration files, results stored in OSGeo SVN:
http://svn.osgeo.org/osgeo/foss4g/benchmarking/

Wiki home: http://wiki.osgeo.org/wiki/Benchmarking_2009

Mailing list: http://lists.osgeo.org/mailman/listinfo/benchmarking

Additional MapServer Results (not presented)

MapServer Roads/Lines Test Results

MapServer Labelled Roads/Lines Test Results

MapServer Polygon Test Results

MapServer Point Test Results

MapServer Raster Test Results

FastCGI Configuration Used for MapServer Tests

FastCgiServer /var/www/cgi-bin/mapserv.fcgi -processes 8 -initial_env LD_LIBRARY_PATH=/usr/local/lib:/opt/build/instant_client_11_1/lib:/opt/build/arcsde_c_sdk/lib

GeoServerMapServer FCGI

110.19.6

1024.523.6

2024.623.9

4023.924.5

ThreadsGeoServerMapServer FCGI

11110.8

1025.124.3

2025.324.6

4025.825

???Pagina ??? (???)10/22/2009, 13:28:43Pagina / ThreadsGeoServerMapServer FCGI

110.19.6

1024.523.6

2024.623.9

4023.924.5

???Pagina ??? (???)10/22/2009, 13:41:12Pagina / GeoServerMapServer FCGI

19.58.9

1023.322.3

2023.623.2

4023.323.6

Line DrawingMaps/secThroughput - 1Throughput - 10Throughput - 20Throughput - 40

shape5.111.311.111.5

shape 54 fcgi8.818.518.319.1

shape 56 fcgi8.217.116.817.6

pg4.411.21111.7

pg fcgi6.515.515.115.2

oracle2.47.77.78.7

oracle fcgi5.612.812.212.9

sde16.77.27.3

sde fcgi4.710.99.811

17.26.6

1016.315.8

2014.815.1

4016.616.4

17.26.6

1016.315.8

2014.815.1

4016.616.4

15.95.6

1012.512.6

2011.511.6

4012.712.8

15.95.6

1012.512.6

2011.511.6

4012.712.8

16.14.7

1013.311.1

2011.69.7

401311

16.14.7

1013.311.1

2011.69.7

401311

???Pagina ??? (???)10/22/2009, 13:47:32Pagina / GeoServerMapServer CGIMapServer FCGI

16.35.46.6

1013.411.914.5

2013.912.715.5

4013.212.615.3

LabelGeoServerMapServer CGIMapServer FCGI

16.35.46.6

1013.411.914.5

2013.912.715.5

4013.212.615.3

1014.318.6

2014.919.4

4014.719.1

1013.216.8

2013.917.5

4013.517.3

1013.216.8

2013.917.5

4013.517.3

15.66.4

1012.816.4

2013.317.4

4013.117.2

15.66.4

1012.816.4

2013.317.4

4013.117.2

1547.4

1011.112.620

2011.212.819.8

4010.512.619.1

ThreadsGeoServerMapServer CGIMapServer FCGI

1547.4

1011.112.620

2011.212.819.8

4010.512.619.1

1116.512

102514.925.8

2025.215.326.2

4025.115.526.8

GeoServerMapServer CGIMapServer FCGI

19.36.612.2

1024.314.926.5

202515.326.6

4025.215.727.1

110.710.1

1024.423.7

2025.224.2

4025.724.5

110.29.2

1023.121.1

2023.821.7

402322

110.29.2

1023.121.1

2023.821.7

402322

19.58.9

1023.322.3

2023.623.2

4023.323.6

110.59.8

102523.3

2025.223.4

4024.623.9

11110.8

1025.124.3

2025.324.6

4025.825

110.59.8

102523.3

2025.223.4

4024.623.9

14.82.76.5

109.71114.9

209.611.416.4

409.911.316.9

14.82.76.5

109.71114.9

209.611.416.4

409.911.316.9

16.44.811.1

1013.713.326.9

2013.31528.1

4013.615.227.3

16.44.811.1

1013.713.326.9

2013.31528.1

4013.615.227.3

???Pagina ??? (???)24/10/2009, 12:33:20Pagina / ThreadsGeoServerMapServer CGIMapServer FCGI

1116.512

102514.925.8

2025.215.326.2

4025.115.526.8

110.710.1

1024.423.7

2025.224.2

4025.724.5

???Pagina ??? (???)10/23/2009, 12:01:33Pagina / Lines with LabelsMaps/secThroughput - 1Throughput - 10Throughput - 20Throughput - 40

shape4.410.210.510.4

shape 54 fcgi9.220.52120.8

shape 56 fcgi6.614.515.515.3

pg4.210.510.610.7

pg fcgi614.114.614.4

oracle2.88.99.39.4

oracle fcgi5.412.913.513.3

oracle ogr0.71.#NAN1.#NAN1.#NAN

sde17.79.28.8

sde fcgi512.913.413.3

19.36.612.2

1024.314.926.5

202515.326.6

4025.215.727.1

17.658.2

10151117.1

2013.11116.8

4015.111.517.6

LabelGeoServerMapServer CGIMapServer FCGI

17.658.2

10151117.1

2013.11116.8

4015.111.517.6

???Pagina ??? (???)10/22/2009, 13:36:47Pagina /

WMS Performance Shootout 2009

Technology

POST Stadium Shootout

Teste de performance de WMS entre GeoServer e Mapserver

Custom Shootout 2013

Shootout 2014 final

Evil Clown Shootout

Six Gun Shootout

High Performance WMS with Virtualization - hnee.de · HIGH PERFORMANCE WMS WITH ... Monitoring Distributed Packet from DB ... • VMware® vCenter™ Single Sign-On Server Technical

WPS Shootout

Wild Wild West Shootout

Shootout 2013

San Juan Shootout.- Redacted

Performance and statistical analysis of WMS serversblog.sourcepole.com/assets/2010/9/10/foss4g2010_wms.pdf · FOSS4G Barcelona 8.9.10 Performance and statistical analysis of WMS servers

Wms Performance Tests Map Server Vs Geo Server

z/OS Performance Monitoring Shootout ASG, BMC, CA and IBM€¦ · z/OS Performance Monitoring Shootout ASG, BMC, CA and IBM Gary Henderson – Director of Product Management ASG-

Texas Shootout Notes

WMS Performance Shootout 2010

Tablet Shootout

6 l Shootout Report

Optimizing Supply Chain Performance WMS Solution …savantwms.com/Documents/Savant - WMS Overview Web Version.pdf · Optimizing Supply Chain Performance. ... moves, picking and shipping

2014 TEAM SHOOTOUT