Advanced Map Caching TopicsTecnicas avanzadas de almacenamiento en cache de mapasTecnicas avanzadas de almacenamiento en cache de mapas

Esri European User Conference 2011

Prerequisite knowledge

• What is map caching?

• How to author a map for caching

• How to build map cache tiles

Topics in this session

• What’s new in caching in ArcGIS 10

• Approaching a large caching job- Building a test caches

- Choosing the tiles to cache

- Monitoring the tile creation

- Case study: State of Indiana imagery basemap

• Cache update strategies

What’s new in caching in ArcGIS 10ArcGIS 10

Mixed mode image format

• Creates JPEG tiles unless transparent pixels detected

• If transparent pixels detected, • If transparent pixels detected, creates PNG32

• Enormous space savings for large caches

Mixed format helps with this JPEG on JPEG cache overlay problem

How the mixed format creates tiles

Compact cache storage format

• Stores tiles in compact, continuous file streams (“bundles”)

- Maximum ~16,000 tiles per bundle

• Faster copying

• Smaller size on disk

• Generally faster to create

• Less fragmentation

Move spatial subsets of tiles between caches

• New export and import tools allow collaborative and “best available” caching

Export Import

Cache export tool

• Export Map Server Cache- Based on extent or polygon features

- Convert storage format

- Use for cache import or as a disconnected cache

Exported using Nevada and Utah state boundary features.

Export

Cache import tool

• Import Map Server Cache- Based on extent or polygon features

- Must have same storage format

Import from a previously exported map cache.

Import

Cache import for imagery

• Merge high resolution imagery into a base cache

• Seamless integration import feature boundary

Collaborative caching

• Use export and import tools - Import the “best available” cached tiles

- Esri Community Base Maps program

PasadenaCity CollegeContribution

Caches treated as a raster dataset in ArcGIS

• Add Data and browse to cache directory.- Looks like any other raster

• Export caches for disconnected field work• Export caches for disconnected field work- Export to compact format recommended

Approaching a large caching job

What should you cache?

• Base maps

• Operational layers that satisfy one of the followin g:- High volumes of traffic

- Don’t change often

- Cover small scales only

Choosing scales

• Build just the scales you need

• Determine closest scale (Raster resolution)

Sample 10 level cache

• Divide scale by 2 for each subsequent scale

• Adjust smallest scale to full extent

• Consider Web Mercator scales

Final level is ~75% of the total

Choosing an image format

• Image format affects- Tile storage space requirements

- Web application performance (speed and supported browsers)

- Tile image quality- Tile image quality

- Tile transparency

Basemap: Image format guidelines

• Fewer colors (~256)- PNG 8

• Many colors• Many colors- JPEG

- Small file size

- Quality settings range from 55 (orthos) - 90 (vector maps)

Choosing JPEG quality

• Which one looks better?

JPEG 55 – 15KB JPEG 90 – 33KBJPEG 90 – 21KB JPEG 55 – 10KB

Operational layers: image format guidelines

• Vector overlays (roads, boundaries)- PNG 32 because of good antialiasing

• Other overlays (weather, etc)• Other overlays (weather, etc)- MIXED or PNG (bit depth depends on number of colors )

Example: Tiles are too large

Aerial photo and Aerial photo and vector blend using vector blend using

PNG 32PNG 32

Antialiasing: Often worth the cost

Apply antialiasing in the MSD for best performance

1. 2.

Box already checked shows that antialiasing is happeningthrough MSD ☺

Gives you this…

Reducing duplicate labels

• Caches drawn using supertiles

• Duplicates can occur across supertileboundaries

SupertileNo antialiasing4096x4096

• Solution is annotationSupertileantialiasing2048x2048

512x512 tile size

256x256 tile sizeArcGIS Online / Bing / Google

B u i l d i n g a t e s t c a c h e

How to build a test cache

1. Select test area with varying geography

2. Create a simple feature class covering the test areacovering the test area

3. Create tiles at all scale levels based on this feature class

Note the following from the test cache

• Appearance of tiles

• Performance of tiles in client

• Cache creation time

• Cache size on disk

Choosing the tiles to cache

Not all tiles are created equal

http://hotmap.msresearch.us

Use GIS modeling to predict frequently visited area s

This feature class covers 25% of California’s area but includes 97% of its population

Tiles you don’t want to create and store

• Cache on demand

• Insert a “Data not available” tile

OR

Make your own heatmap of tile usage: ArcGIS Server descriptive tile analysis

Caching based on a feature class

• Features should be:- Big

- Generalized

Why do the features need to be big?

• How do I find out the bundle size?

- Map Server Cache Tiling Scheme to Polygons tool

- See Esri Dev Summit 2011 - See Esri Dev Summit 2011 presentation online for full demo

Monitoring the tile creation

Setting the number of instances

• Keep your CPU below 100%• Start with N+1 instances where N = the number of co res

on server

• Adjust number of instances based on test cache

Manage Map Server Cache TilesTask Manager

2 CPU’s3 Instances

Generate using multiple LAN hosts

• Configure instances for CPU’s on all SOC hosts

• Enable “Use local cache directory…”

SOC hosts

SharedCache

Temp cache

Temp cache

SOMhosts cache

Temp cache

Use status tracking with your features

Helpful script:

Create gridded feature class for tracking ArcGIS Server map caching jobs

Recovering from errors

• Generate rebuild features from error log- Convert map cache-build log failures to footprint

polygons tool

• Cache Validation toolLooks for missing tiles- Looks for missing tiles

- Looks for tiles below a certain size (white or gray tiles)- False positives with building roofs and water

- Useful for validating source imagery

Case study: State of Indiana imagery basemapimagery basemap

Requirements

• Single Indiana state basemap with most recent data- Maintain 6” resolution where available

• ArcGIS Online/Bing/Google tiling scheme

• Integrate neighboring data for small scales• Integrate neighboring data for small scales

• Record metadata for imagery in basemap

Approach

6" scale 576.00 20 Individual 6" counties

1,128.50 19

Use gridded tile2,256.99 18

4,513.99 17

9,027.98 16

Clip to state boundary

18,055.96 15

36,111.91 14

72,223.82 13

144,447.64 12

Web

Mer

cato

r Ti

ling

Sch

eme

144,447.64 12

288,895.29 11

IN Full extent

577,790.58 10

1,155,581.15 9

2,311,162.31 8

4,622,324.61 7

9,244,649.23 6

18,489,298.45 5

Not Built

36,978,596.91 4

73,957,193.82 3

147,914,387.60 2

295,828,775.30 1

591,657,550.50 0

Web

Mer

cato

r Ti

ling

Sch

eme

Stats on the final cache

• Image format: JPEG 55

• 62,245,522 tiles

• Total cache size 452.72 GB

• Average tile size 7.63 KB• Average tile size 7.63 KB

• 3 weeks to build

C a c h e u p d a t e s t r a t e g i e s

Can you update the cache?

• How current does the data need to be?

• How fast can the update take place?

• Can the update be scripted?

• Esri Spain elections example

Strategic cache updates

1. Detect where changes have taken place

2. Merge these features

3. Use them to define the update area

Helpful scripts

•Show edits since reconcile

•Compare two feature classes in a file geodatabase

Production ArcGIS Server Instance

Staging ArcGIS Server Instance

Map service

Map serviceAll layers for All layers for

cartography of cartography of map servicemap service

Layers for TOC Layers for TOC and Queryand Query

Updating a cache using a staging server

Map service

Cache folderCache folder Copy tiles

(On-demand caching needs the full map to build the cache)

Production ArcGIS Server Instance

Staging ArcGIS Server Instance

All layers for All layers for cartography of cartography of

map servicemap service Layers for TOC Layers for TOC and Queryand QueryCopy tiles

Updating a cache and data using a staging server

Map serviceMap service

Cache folderCache folder

Multi-user versioned Geodatabase Geodatabase (File or Multi-user)

GeodatabaseReplication

Q u e s t i o n s