Colorado School of Mines Department of Electrical Engineering and Computer Science
Colorado School of Mines
Image and Multidimensional Signal
Processing
Professor William Hoff
Department of Electrical Engineering and Computer
Science
http://inside.mines.edu/~whoff/
Colorado School of Mines Department of Electrical Engineering and Computer Science
Spatial Filtering
2
Colorado School of Mines Department of Electrical Engineering and Computer Science Colorado School of Mines Department of Electrical Engineering and Computer Science
Spatial filtering
• Main idea
– Define a neighborhood of a pixel
– Define an operation on pixels in the neighborhood
– Output pixel value = result of the operation
• Topics
– Linear filtering
• Correlation, convolution
• Normalized cross correlation
– Important examples
• Smoothing filters (box average, Gaussian)
• Sharpening filters (Laplacian, Sobel)
– Non-linear filters
• Order-statistics filters (primarily median filter)
3
Colorado School of Mines Department of Electrical Engineering and Computer Science
Linear Spatial Filtering
• Create a filter or mask w, size m x n
• Apply to image f, size M x N
• Compute the sum of products of mask coefficients with
corresponding pixels under mask
• Slide mask over image, apply at each point
• It is a linear operation
/2 /2
/2 /2
( , ) ( , ) ( , )
( , ) ( , )
m n
s m t n
g x y w s t f x s y t
w x y f x y
( , ) ( , ) ( , ) ( , )
( , ) ( , ) ( , ) ( , ) ( , ) ( , ) ( , )
w x y a f x y a w x y f x y
w x y f x y g x y w x y f x y w x y g x y
4
Also called “cross-
correlation”
Colorado School of Mines Department of Electrical Engineering and Computer Science Colorado School of Mines Department of Electrical Engineering and Computer Science
2/
2/
2/
2/
),(),(),(m
ms
n
nt
tysxftswyxg
5
Colorado School of Mines Department of Electrical Engineering and Computer Science Colorado School of Mines Department of Electrical Engineering and Computer Science
Example
• 3x3 averaging (“box”) filters
• Do manual calculation on a
corner
6
0 0 0 0 0 0 0 ..
0 0 0 0 0 0 0 ..
0 0 0 0 0 0 0 ..
0 0 0 9 9 9 9 ..
0 0 0 9 9 9 9 ..
0 0 0 9 9 9 9 ..
0 0 0 9 9 9 9 ..
: : : : : : : ..
( , ) ( , ) ( , )g x y w x y f x y ( , )f x y
Colorado School of Mines Department of Electrical Engineering and Computer Science Colorado School of Mines Department of Electrical Engineering and Computer Science
Example
• 3x3 averaging filters
• Manual calculation on
corner
7
Colorado School of Mines Department of Electrical Engineering and Computer Science Colorado School of Mines Department of Electrical Engineering and Computer Science
Matlab Examples
• fspecial
– useful to create filters
• imfilter
– does cross correlation
8
clear all
close all
% Synthetic image of a white square
I = zeros(200,200);
I(50:150, 50:150) = 1;
imshow(I,[]);
% Apply 3x3 box filter
w = [ 1 1 1; 1 1 1; 1 1 1]/9;
I2 = imfilter(I, w);
imtool(I2,[]);
% Apply 15x15 box filter
w = fspecial('average', [15 15]);
I3 = imfilter(I, w);
imtool(I3,[]);
% Show that averaging reduces noise
I = I + randn(200,200)*0.5;
imshow(I,[]);
w = fspecial('average', [5 5]);
I4 = imfilter(I, w);
figure, imshow(I4,[]);
Colorado School of Mines Department of Electrical Engineering and Computer Science
Sharpening Spatial Filters – First Derivative
• First derivative
– In discrete form (can also do central difference)
– As a filter
• Sobel operators – combine smoothing with derivative
0
( , ) ( , ) ( , )lim
f x y f x y f x y
x
( , )( 1, ) ( , )
f x yf x y f x y
x
-1 +1
( 1) ( 1) / 2f
f x f xx
-.5 0 +.5
-1 0 1
-2 0 2
-1 0 1
-1 -2 -1
0 0 0
1 2 1
dx filter dy filter
9
Colorado School of Mines Department of Electrical Engineering and Computer Science
Example
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 1 1 1 1 0 0
0 0 1 1 1 1 0 0
0 0 1 1 1 1 0 0
0 0 1 1 1 1 0 0
0 0 1 1 1 1 0 0
-1 0 1
-2 0 2
-1 0 1
( , ) ( , ) ( , )g x y w x y f x y
( , )w x y
( , )f x y
10
Colorado School of Mines Department of Electrical Engineering and Computer Science
Example
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 1 1 1 1 0 0
0 0 1 1 1 1 0 0
0 0 1 1 1 1 0 0
0 0 1 1 1 1 0 0
0 0 1 1 1 1 0 0
-1 0 1
-2 0 2
-1 0 1
( , ) ( , ) ( , )g x y w x y f x y
( , )w x y
( , )f x y
11
Colorado School of Mines Department of Electrical Engineering and Computer Science Colorado School of Mines Department of Electrical Engineering and Computer Science
Gradient
yf
xff
• Compute gradient components using first derivative operators
• Gradient magnitude indicates the location of edges in the image
• Gradient direction shows direction of edge
• We can use the Sobel operators for df/dx, df/dy
2/122
y
f
x
ff
1tanf y
f x
12
Colorado School of Mines Department of Electrical Engineering and Computer Science Colorado School of Mines Department of Electrical Engineering and Computer Science
13
Matlab example
• Note - the result is not quite
correct
• We should first convert the
input image to type “double”
before applying “imfilter”
(why?)
clear all
close all
I = imread('moon.tif');
imshow(I,[]);
% Create Sobel filters
hy = [ % can also do hy = fspecial('sobel');
-1 -2 -1;
0 0 0;
+1 +2 +1];
hx = hy';
Ix = imfilter(I, hx);
Iy = imfilter(I, hy);
figure;
subplot(1,2,1), imshow(Ix,[]);
subplot(1,2,2), imshow(Iy,[]);
Colorado School of Mines Department of Electrical Engineering and Computer Science Colorado School of Mines Department of Electrical Engineering and Computer Science
14
Matlab example (continued)
>> help atan2
atan2 Four quadrant inverse tangent.
atan2(Y,X) is the four quadrant arctangent of the real parts of the
elements of X and Y. -pi <= atan2(Y,X) <= pi.
See also atan.
Reference page in Help browser
doc atan2
% Compute gradient magnitude (note the use of ".")
Ig = (Ix.^2 + Iy.^2) .^ 0.5;
figure, imshow(Ig, []);
% Compute gradient direction
Iang = atan2(Iy,Ix);
figure, imshow(Iang, []);
% Only display angles where gradient magnitude is large
Mask = (Ig > 100);
figure, imshow(double(Mask) .* Iang, []);
colormap jet
Colorado School of Mines Department of Electrical Engineering and Computer Science
Sharpening Spatial Filters – 2nd derivative
• Second derivative
– As a filter
• Two dimensional – the Laplacian
– As a filter
2
2
( , ) ( , )( 1, ) 2 ( , ) ( 1, )
f x y f x yf x y f x y f x y
x x x
+1 -2 +1
2
2
2
22
y
f
x
ff
0 +1 0
+1 -4 +1
0 +1 0
15
0 0 0
+1 -2 +1
0 0 0
0 +1 0
0 -2 0
0 +1 0
2
2
x
2
2
x
= +
Colorado School of Mines Department of Electrical Engineering and Computer Science
High Boost Filtering
• We combine original image with results of Laplacian – result is
the original image with enhanced detail
– Also called unsharp masking
• In general
2( , ) ( , )g x y f x y f 0 -1 0
-1 5 -1
0 -1 0
16
),(),(),( yxfyxAfyxg
Colorado School of Mines Department of Electrical Engineering and Computer Science Colorado School of Mines Department of Electrical Engineering and Computer Science
17
Colorado School of Mines Department of Electrical Engineering and Computer Science Colorado School of Mines Department of Electrical Engineering and Computer Science
Vector representation of correlation
• Correlation is a sum of products of corresponding terms
• We can think of correlation as a dot product of vectors w and f
• If images w and f are similar, their vectors are aligned
1 1 2 2
1
mn mn
mnT
k k
k
c w f w f w f
w f
w f
/2 /2
/2 /2
( , ) ( , ) ( , )
( , ) ( , )
m n
s m t n
c x y w s t f x s y t
w x y f x y
18
Colorado School of Mines Department of Electrical Engineering and Computer Science Colorado School of Mines Department of Electrical Engineering and Computer Science
Correlation can do matching
• Let w(x,y) be a template sub-image
• Try to find an instance of w in image f(x,y)
• The correlation score c(x,y) is high where w matches f
/2 /2
/2 /2
( , ) ( , ) ( , )
( , ) ( , )
m n
s m t n
c x y w s t f x s y t
w x y f x y
19
Colorado School of Mines Department of Electrical Engineering and Computer Science Colorado School of Mines Department of Electrical Engineering and Computer Science
20
Colorado School of Mines Department of Electrical Engineering and Computer Science Colorado School of Mines Department of Electrical Engineering and Computer Science
Template Matching (continued)
• Since f is not constant everywhere, we need to normalize
• The result is between 0..1 (for positive valued images)
• We can get better precision by subtracting off means
• This result is between -1 .. +1
T
c w f
w f
,
1/2
22
, ,
[ ( , ) ][ ( , ) ]
( , )
[ ( , ) ] [ ( , ) ]
s t
s t s t
w s t w f x s y t f
c x y
w s t w f x s y t f
• This is the
“normalized
correlation
coefficient”
21
22
3
2
2
2
1 where Nwwww w
Colorado School of Mines Department of Electrical Engineering and Computer Science Colorado School of Mines Department of Electrical Engineering and Computer Science
Normalized Cross-correlation
• Normalized cross-correlation (values range from -1 .. +1)
• where
– w(s,t) is a subimage, size mxn
– f(x,y) is a large image, size MxN
– sums are taken over s=1..m, t=1..n
– is the local mean of f, in a mxn neighborhood around (x,y)
,
1/2
22
, ,
[ ( , ) ][ ( , ) ( , )]
( , )
[ ( , ) ] [ ( , ) ( , )]
s t
s t s t
w s t w f x s y t f x y
c x y
w s t w f x s y t f x y
w(s,t) f(x,y)
Size mxn
Size MxN
c(x,y) Correlation
scores
,
1( , ) ( , )
s t
f x y f x s y tmn
( , )f x y
Size MxN 22
See pg. 870 in
the Gonzalez &
Woods textbook
Colorado School of Mines Department of Electrical Engineering and Computer Science Colorado School of Mines Department of Electrical Engineering and Computer Science
Matlab Example
• Crop a small subimage from left image, find its match in the right image
I1 = imread('test000.jpg');
T = imcrop(I1, [66, 213, 86, 233]);
I2 = imread('test012.jpg');
C = normxcorr2(T, I2);
% The scores are in an image that is slightly bigger than the original
% image ... it is expanded by half the size of the template in all
% directions. So we will crop out the center portion.
Csub = imcrop(C, [(size(T,2)-1)/2+1 (size(T,1)-1)/2+1 size(I2,2)-1 size(I2,1)-1]);
cmax = max(Csub(:));
[y x] = find(Csub==cmax);
23
Colorado School of Mines Department of Electrical Engineering and Computer Science Colorado School of Mines Department of Electrical Engineering and Computer Science
Doing cross correlation in Matlab without normxcorr2
• The easiest way is to implement the equation directly, using “for”
loops
• You scan through the image ... at every pixel (x,y):
– Extract the subimage fs(x,y) from the image f surrounding x,y
– Subtract off the local image mean value at (x,y)
– Compute the sum of the product of (fs-fmean) and (w-mean2(w); this is the
numerator
– The denominator can be similarly computed
– Dividing the two gives you the value of the normalized cross correlation at pixel
(x,y) in the ouput
• Note - there are faster ways to do this, using Matlab’s ability to
operate on whole matrices
24
Colorado School of Mines Department of Electrical Engineering and Computer Science Colorado School of Mines Department of Electrical Engineering and Computer Science
Doing cross correlation in Matlab without normxcorr2
(continued)
• Numerator
– where
• Then
• The first term is the correlation of (w-wmean) and f. The second term is zero because
• So in Matlab: N = imfilter(f,w-mean2(w));
,
[ ( , ) ][ ( , ) ( , )]s t
N w s t w f x s y t f x y
is the mean of all pixels in w; it can be computed once. It is a scalar.
( , ) is the local mean of pixels in f, in a mxn window centered at (x,y).
This can be represented as an image of size MxN.
w
f x y
,
, ,
[ ( , ) ][ ( , ) ( , )]
[ ( , ) ] ( , ) ( , ) [ ( , ) ]
s t
s t s t
w s t w f x s y t f x y
w s t w f x s y t f x y w s t w
, , ,
[ ( , ) ] ( , ) 0s t s t s t
w s t w w s t w mnw mnw
25
Colorado School of Mines Department of Electrical Engineering and Computer Science Colorado School of Mines Department of Electrical Engineering and Computer Science
Doing cross correlation in Matlab without normxcorr2
(continued)
• Denominator
• In Matlab, you can compute fmean at every pixel by filtering with a mxn rectangle of 1’s (and dividing by mxn):
[m n] = size(w);
fmean = imfilter(f, ones(m,n))/(m*n);
• The first term is a scalar, and can be computed once. We rewrite the second term:
1/2
22
, ,
[ ( , ) ] [ ( , ) ( , )]s t s t
D w s t w f x s y t f x y
2
,
2 2
, , ,
2 2 2
,
2 2
,
[ ( , ) ( , )]
( , ) 2 ( , ) ( , ) ( , )
( , ) 2( ) ( , ) ( ) ( , )
( , ) ( ) ( , )
s t
s t s t s t
s t
s t
f x s y t f x y
f x s y t f x y f x s y t f x y
f x s y t mn f x y mn f x y
f x s y t mn f x y
This can be computed by filtering again with a mxn rectangle of 1’s; i.e., imfilter(f.^2, ones(m,n)) 26
Colorado School of Mines Department of Electrical Engineering and Computer Science Colorado School of Mines Department of Electrical Engineering and Computer Science
Correlation and Convolution
• Correlation
• Convolution
– Note that
– So to implement convolution we can reflect w(x,y) and then do correlation
/2 /2
/2 /2
( , ) ( , ) ( , )
( , ) ( , )
m n
s m t n
g x y w s t f x s y t
w x y f x y
/2 /2
/2 /2
( , ) ( , ) ( , )
( , ) ( , )
m n
s m t n
g x y w s t f x s y t
w x y f x y
/2 /2 /2 /2
/2 /2 /2 /2
( , ) ( , ) ( , ) ( , )m n m n
s m t n s m t n
w s t f x s y t w s t f x s y t
27
• Convolution is very
similar to
correlation, except
that we flip one of
the functions
• Also a linear
operation
Colorado School of Mines Department of Electrical Engineering and Computer Science Colorado School of Mines Department of Electrical Engineering and Computer Science
1D Examples
28
Figure 3.29
Colorado School of Mines Department of Electrical Engineering and Computer Science Colorado School of Mines Department of Electrical Engineering and Computer Science
1D Examples
29
Colorado School of Mines Department of Electrical Engineering and Computer Science Colorado School of Mines Department of Electrical Engineering and Computer Science
2D Examples
30
Colorado School of Mines Department of Electrical Engineering and Computer Science Colorado School of Mines Department of Electrical Engineering and Computer Science
2D Examples
31
Colorado School of Mines Department of Electrical Engineering and Computer Science Colorado School of Mines Department of Electrical Engineering and Computer Science
Properties
• Convolution is
– Commutative
f * g = g * f
– Associative
f * (g * h) = (f * g) * h
– Distributive
f * (g + h) = f * g + f * h
• Correlation is distributive but not commutative or associative
• Convolution of a filter h(x,y) with an impulse d(x,y) yields h(x,y)
32
Colorado School of Mines Department of Electrical Engineering and Computer Science Colorado School of Mines Department of Electrical Engineering and Computer Science
Gaussian – an important smoothing filter
• 1D
• 2D
2
221
( )2
x
h x e
2 2
222
1( , )
2
x y
h x y e
-10 -8 -6 -4 -2 0 2 4 6 8 100
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0
10
20
30
40
0
10
20
30
40
0
0.002
0.004
0.006
0.008
0.01
Notes:
is the standard
deviation
Values should sum to 1
Mask size should be at
least ±3
33
Colorado School of Mines Department of Electrical Engineering and Computer Science Colorado School of Mines Department of Electrical Engineering and Computer Science
Convolving with two Gaussians
• You can convolve a function with a Gaussian, then convolve again
with a Gaussian. The effect is the same as convolving with a single,
larger Gaussian
• Why? Since convolution is associative
• and if
• then
2 2
2 21 22 2
1 2
1 2
1 1,
2 2
x x
g e g e
2
2 21 22
1 22 2
1 2
1
2
x
g g e
1 2 1 2g g f g g f
i.e., it is a Gaussian
with standard
deviation
2 2
1 2new
34
Colorado School of Mines Department of Electrical Engineering and Computer Science Colorado School of Mines Department of Electrical Engineering and Computer Science
Proof
• You can prove this from the definition of convolution
• And use the idea that
where the last term does not depend on t and the first one is just
another Gaussian, but centered around a different point.
( ) ( )f g f t g t x dt
2 22 2 2 22 /22 2 /2t x t xt t tx x xe e e e e
35
Colorado School of Mines Department of Electrical Engineering and Computer Science Colorado School of Mines Department of Electrical Engineering and Computer Science
Interesting fact
• If you repeatedly convolve an image N times with any
filter, the effect (for large N) is that of convolving with a
Gaussian
• This comes from the Central Limit Theorem: “The sum
of N mutually independent random variables with zero
means and finite variances tends toward the normal
probability distribution”
• And the fact that the pdf of the sum of two independent
random variables is the convolution of their pdf’s
Colorado School of Mines Department of Electrical Engineering and Computer Science
Separable kernels (filter masks)
• If a 2D correlation (or convolution) kernel can be separated into two
1D kernels, the operation is much faster
– If
– then
• Namely, do a 1D correlation with h2(y) along the individual columns
of the input image, then do a 1D correlation with h1(x) along the
rows of the result from the previous step
• If h(x,y) is nxn, and f(x,y) is NxN
– Cost of a 2D correlation?
– Cost of two 1D correlations?
)()(),( 21 yhxhyxh
1 2( , ) ( , ) ( ) ( ) ( , )h x y f x y h x h y f x y
37
Colorado School of Mines Department of Electrical Engineering and Computer Science
Separable kernels (filter masks)
• If a 2D correlation (or convolution) kernel can be separated into two
1D kernels, the operation is much faster
– If
– then
• Namely, do a 1D correlation with h2(y) along the individual columns
of the input image, then do a 1D correlation with h1(x) along the
rows of the result from the previous step
• If h(x,y) is nxn, and f(x,y) is NxN
– Cost of a 2D correlation?
– Cost of two 1D correlations?
)()(),( 21 yhxhyxh
1 2( , ) ( , ) ( ) ( ) ( , )h x y f x y h x h y f x y Prove this
(HW2)
38
Colorado School of Mines Department of Electrical Engineering and Computer Science Colorado School of Mines Department of Electrical Engineering and Computer Science
Nonlinear Filters
• Median filter - a non-linear smoothing filter
• Steps:
– Order (sort) pixel values within neighborhood
– Replace center with some value based on the ordering (i.e., median)
• Advantages
– Especially good for reducing impulse, or shot-and-pepper noise
– Doesn’t blur sharp edges in the image
1D noisy step edge median filtered, filter width=7 0 5 10 15 20 25 30 35 40 45 50
-2
0
2
4
6
8
10
12
0 5 10 15 20 25 30 35 40 45 500
2
4
6
8
10
12
Matlab
functions
medfilt1,
medfilt2,
ordfilt2
39
Colorado School of Mines Department of Electrical Engineering and Computer Science Colorado School of Mines Department of Electrical Engineering and Computer Science
Example on ramp edge
1: 4,6 :10( )
20 5
x xf x
x
-5 0 5 10 15
0
5
10
15
20
25
40
Colorado School of Mines Department of Electrical Engineering and Computer Science Colorado School of Mines Department of Electrical Engineering and Computer Science
Example on ramp edge
1: 4,6 :10( )
20 5
x xf x
x
-5 0 5 10 15
0
5
10
15
20
25
41
Colorado School of Mines Department of Electrical Engineering and Computer Science Colorado School of Mines Department of Electrical Engineering and Computer Science
42
Colorado School of Mines Department of Electrical Engineering and Computer Science Colorado School of Mines Department of Electrical Engineering and Computer Science
Examples - Matlab
• Read image “eight.tif”
– Add noise: imnoise
– Compare imfilter, medfilt2
43
clear all
close all
I = imread('eight.tif');
imshow(I,[]);
Inoisy = imnoise(I, 'salt & pepper', 0.05);
imtool(Inoisy,[]);
I1 = imfilter(Inoisy, fspecial('average'));
imshow(I1, []);
I2 = medfilt2(Inoisy);
imshow(I2, []);
Colorado School of Mines Department of Electrical Engineering and Computer Science Colorado School of Mines Department of Electrical Engineering and Computer Science
Summary
• A spatial filter operates on a neighborhood of a pixel.
The output pixel value is the result of the operation.
• A “linear” spatial filter performs a sum of products of the
neighborhood values with the corresponding filter values.
• The median filter is an example of a “non-linear” filter.
• Filters can be used for smoothing and sharpening.
44
Colorado School of Mines Department of Electrical Engineering and Computer Science Colorado School of Mines Department of Electrical Engineering and Computer Science
Questions
• Is the 2D Gaussian filter separable? Why or
why not?
• What is the difference between convolution and
correlation?
• Why might the median filter be preferable for
some situations?
45