Raster Processing with Scipy.ndimage (Dev Meet Up II)

Raster processing with scipy.ndimageHenry Walshawhenry@pythoncharmers.com @om_hennersN

ggggggggggggggiiiiiiiiisss....ssttttttaaccccccccccccccckkkkkkkkkkkkkkeeexxxxxcccccccccccchhhhaaaaaaannnnngggggggggggggggggg

eeee..ccccoooo

mmmmmmmmmmmmmmmmmm

cccccccccccchhhhhhhhhhhhxxxxxxxxxxxxxx aaaaaaaaaaeeeeeee nnnnnnnnnnnkkkkkkkkkkkkkkkkkkkkkk gggggggggggggggggggggggggggggggggccccccccccccccccccccccccaaaaaaaaa eeeeeeettttttttttttt ....sssssss cccccccccc.... oooooooooossssssssiiiiiiiiiiiii mmmmmmmmmmmmmmmmmiiiiigggggggggggggg

Getting set up

arcpy for ArcGIS 10 requires numpy 1.3.0

This means we’re restricted to scipy 0.7.1, matplotlib 1.0.1 and PIL 1.1.7 (which is still the latest version)

If you’re not using ArcGIS, or you’re using virtualenv get the latest versions!

All code for this talk can be downloaded from github:https://github.com/om-henners/ndimage_talk.git

Why process with scipy?

Open Source scientific algorithms

Easy to set up for large concurrent processing on local PCs and in the cloud (see PiCloud)

It’s free!*

*Well, aside from development cost

Getting data in and out

We’ll be using the arcpy.RasterToNumPyArray and arcpy.NumPyArrayToRaster functions

Alternatives include GDAL Python bindings, scipy image read functions, and many others

See getting_data_in.py and getting_data_out.py

Calculating median filter

Once you’ve got an array it’s one line of code to perform a basic filter function

scipy.ndimage.filters.median_filter(a, size=9)

Getting more complex

n-dimensional processing

We can supply a size or filter shape, but now we have to be even more aware of edge effects

Still the same one line of code

Generic filters in scipy

Most Spatial Analyst operations are in ndimage

For everything else there’s generic_filter

Flattens the target region and passes through to a callback function

Can be used to handle null data as ndimage can’t handle masked arrays (yet)

Calculating slope

We can implement a simple slope calculation using generic filter over a 3x3 footprint

We can use the form as described by the ESRI documentation:

slope = e is atan ( sqrt( [dz/dx]**2 + [dz/dy]**2 ) ) where[dz/dx] = ((c + 2f + i) - (a + 2d + g) / (8 * x_cellsize)[dz/dy] = ((g + 2h + i) - (a + 2b + c)) / (8 * y_cellsize))

Slower than a standard ndimage filter, but faster thanarcpy!

Calculating a variable focal maximum

Calculate the focal maximum for every point in an array, with the focal annulus defined by a radius from another raster

Don’t use a generic filter (there’s no need)

Don’t use ArcGIS Spatial analyst (it’s really slow)

Method 100x100 random raster

generic filter 4.7 secs

arcpy Spatial Analyst 6.7 secs

Do use inbuilt functions

It’s faster to calculate the focal sum using ndimage.maximum_filter for every possible buffer value than to roll your own function

100x100 array? 0.009 seconds

Last words on scipy.ndimage

It’s awesome, and free and threadsafe

It ties into any number of other packages (scikits-learn, scikits-image)

You can do just about anything in ndimage that you can in Spatial Analyst

If you have time to code it

And code it right

John Hunter (1968 - 2012)

http://numfocus.org/johnhunter/