Image filtering Image filtering: compute function of local
neighborhood at each position Really important! Enhance images
Denoise, resize, increase contrast, etc. Extract information from
images Texture, edges, distinctive points, etc. Detect patterns
Template matching
What does it do? Replaces each pixel with an average of its
neighborhood Achieve smoothing effect (remove sharp features) 111
111 111 Slide credit: David Lowe (UBC) Box Filter
Slide 15
Smoothing with box filter
Slide 16
Practice with linear filters 000 010 000 Original ? Source: D.
Lowe
Slide 17
Practice with linear filters 000 010 000 Original Filtered (no
change) Source: D. Lowe
Slide 18
Practice with linear filters 000 100 000 Original ? Source: D.
Lowe
Slide 19
Practice with linear filters 000 100 000 Original Shifted left
By 1 pixel Source: D. Lowe
Slide 20
Practice with linear filters Original 111 111 111 000 020 000 -
? (Note that filter sums to 1) Source: D. Lowe
Slide 21
Practice with linear filters Original 111 111 111 000 020 000 -
Sharpening filter - Accentuates differences with local average
Source: D. Lowe
Slide 22
Sharpening Source: D. Lowe
Slide 23
Other filters 01 -202 01 Vertical Edge (absolute value)
Sobel
Slide 24
Other filters -2 000 121 Horizontal Edge (absolute value)
Sobel
Slide 25
Filtering vs. Convolution 2d filtering 2d convolution
f=imageg=filter
Slide 26
Key properties of linear filters Linearity: filter(f 1 + f 2 )
= filter(f 1 ) + filter(f 2 ) Shift invariance: same behavior
regardless of pixel location filter(shift(f)) = shift(filter(f))
Any linear, shift-invariant operator can be represented as a
convolution Source: S. Lazebnik
Slide 27
More properties Commutative: a * b = b * a Conceptually no
difference between filter and signal But particular filtering
implementations might break this equality Associative: a * (b * c)
= (a * b) * c Often apply several filters one after another: (((a *
b 1 ) * b 2 ) * b 3 ) This is equivalent to applying one filter: a
* (b 1 * b 2 * b 3 ) Distributes over addition: a * (b + c) = (a *
b) + (a * c) Scalars factor out: ka * b = a * kb = k (a * b)
Identity: unit impulse e = [0, 0, 1, 0, 0], a * e = a Source: S.
Lazebnik
Slide 28
Weight contributions of neighboring pixels by nearness 0.003
0.013 0.022 0.013 0.003 0.013 0.059 0.097 0.059 0.013 0.022 0.097
0.159 0.097 0.022 0.013 0.059 0.097 0.059 0.013 0.003 0.013 0.022
0.013 0.003 5 x 5, = 1 Slide credit: Christopher Rasmussen
Important filter: Gaussian
Slide 29
Smoothing with Gaussian filter
Slide 30
Smoothing with box filter
Slide 31
Gaussian filters Remove high-frequency components from the
image (low-pass filter) Images become more smooth Convolution with
self is another Gaussian So can smooth with small-width kernel,
repeat, and get same result as larger-width kernel would have
Convolving two times with Gaussian kernel of width is same as
convolving once with kernel of width 2 Separable kernel Factors
into product of two 1D Gaussians Source: K. Grauman
Slide 32
Separability of the Gaussian filter Source: D. Lowe
Slide 33
Separability example * * = = 2D convolution (center location
only) Source: K. Grauman The filter factors into a product of 1D
filters: Perform convolution along rows: Followed by convolution
along the remaining column:
Slide 34
Separability Why is separability useful in practice?
Slide 35
Practical matters What about near the edge? the filter window
falls off the edge of the image need to extrapolate methods: clip
filter (black) wrap around copy edge reflect across edge Source: S.
Marschner
Slide 36
Practical matters What is the size of the output? MATLAB:
filter2(g, f, shape) shape = full: output size is sum of sizes of f
and g shape = same: output size is same as f shape = valid: output
size is difference of sizes of f and g f gg gg f gg g g f gg gg
full samevalid Source: S. Lazebnik
Slide 37
2006 Steve Marschner 37 Median filters A Median Filter operates
over a window by selecting the median intensity in the window. What
advantage does a median filter have over a mean filter? Is a median
filter a kind of convolution? Slide by Steve Seitz
Slide 38
2006 Steve Marschner 38 Comparison: salt and pepper noise Slide
by Steve Seitz
Slide 39
Why does the Gaussian give a nice smooth image, but the square
filter give edgy artifacts? GaussianBox filter
Slide 40
Hybrid Images A. Oliva, A. Torralba, P.G. Schyns, Hybrid
Images, SIGGRAPH 2006 Hybrid Images,
Slide 41
Why do we get different, distance-dependent interpretations of
hybrid images? ? Slide: Hoiem
Slide 42
Why does a lower resolution image still make sense to us? What
do we lose? Image:
http://www.flickr.com/photos/igorms/136916757/http://www.flickr.com/photos/igorms/136916757/Slide:
Hoiem
Slide 43
Thinking in terms of frequency
Slide 44
Jean Baptiste Joseph Fourier (1768-1830) had crazy idea (1807):
Any univariate function can be rewritten as a weighted sum of sines
and cosines of different frequencies. Dont believe it? Neither did
Lagrange, Laplace, Poisson and other big wigs Not translated into
English until 1878! But its (mostly) true! called Fourier Series
there are some subtle restrictions...the manner in which the author
arrives at these equations is not exempt of difficulties and...his
analysis to integrate them still leaves something to be desired on
the score of generality and even rigour. Laplace Lagrange
Legendre
Slide 45
A sum of sines Our building block: Add enough of them to get
any signal g(x) you want!
Slide 46
Frequency Spectra example : g(t) = sin(2f t) + (1/3)sin(2(3f)
t) = + Slides: Efros
Slide 47
Frequency Spectra
Slide 48
= + =
Slide 49
= + =
Slide 50
= + =
Slide 51
= + =
Slide 52
= + =
Slide 53
=
Slide 54
Example: Music We think of music in terms of frequencies at
different magnitudes Slide: Hoiem
Slide 55
Other signals We can also think of all kinds of other signals
the same way xkcd.com
Slide 56
Fourier analysis in images Intensity Image Fourier Image
http://sharp.bu.edu/~slehar/fourier/fourier.html#filtering
Slide 57
Signals can be composed +=
http://sharp.bu.edu/~slehar/fourier/fourier.html#filtering More:
http://www.cs.unm.edu/~brayer/vision/fourier.html
Slide 58
Fourier Transform Fourier transform stores the magnitude and
phase at each frequency Magnitude encodes how much signal there is
at a particular frequency Phase encodes spatial information
(indirectly) For mathematical convenience, this is often notated in
terms of real and complex numbers Amplitude:Phase:
Slide 59
Computing the Fourier Transform Continuous Discrete k =
-N/2..N/2 Fast Fourier Transform (FFT): NlogN
Slide 60
The Convolution Theorem The Fourier transform of the
convolution of two functions is the product of their Fourier
transforms Convolution in spatial domain is equivalent to
multiplication in frequency domain!
Slide 61
Properties of Fourier Transforms Linearity Fourier transform of
a real signal is symmetric about the origin The energy of the
signal is the same as the energy of its Fourier transform See
Szeliski Book (3.4)
Slide 62
Filtering in spatial domain 01 -202 01 * =
Slide 63
Filtering in frequency domain FFT Inverse FFT = Slide:
Hoiem
Slide 64
Why does the Gaussian give a nice smooth image, but the square
filter give edgy artifacts? GaussianBox filter Filtering
Slide 65
Gaussian
Slide 66
Box Filter
Slide 67
Why does a lower resolution image still make sense to us? What
do we lose? Image:
http://www.flickr.com/photos/igorms/136916757/http://www.flickr.com/photos/igorms/136916757/
Sampling
Slide 68
Throw away every other row and column to create a 1/2 size
image Subsampling by a factor of 2
Slide 69
1D example (sinewave): Source: S. Marschner Aliasing
problem
Slide 70
Source: S. Marschner 1D example (sinewave): Aliasing
problem
Slide 71
Sub-sampling may be dangerous. Characteristic errors may
appear: Wagon wheels rolling the wrong way in movies Checkerboards
disintegrate in ray tracing Striped shirts look funny on color
television Source: D. Forsyth Aliasing problem
Slide 72
Aliasing in video Slide by Steve Seitz
Slide 73
Source: A. Efros Aliasing in graphics
Slide 74
Sampling and aliasing
Slide 75
When sampling a signal at discrete intervals, the sampling
frequency must be 2 f max f max = max frequency of the input signal
This will allows to reconstruct the original perfectly from the
sampled version good bad vvv Nyquist-Shannon Sampling Theorem
Slide 76
Anti-aliasing Solutions: Sample more often Get rid of all
frequencies that are greater than half the new sampling frequency
Will lose information But its better than aliasing Apply a
smoothing filter
Slide 77
Algorithm for downsampling by factor of 2 1.Start with image(h,
w) 2.Apply low-pass filter im_blur = imfilter(image,
fspecial(gaussian, 7, 1)) 3.Sample every other pixel im_small =
im_blur(1:2:end, 1:2:end);
Slide 78
Anti-aliasing Forsyth and Ponce 2002
Slide 79
Subsampling without pre-filtering 1/4 (2x zoom) 1/8 (4x zoom)
1/2 Slide by Steve Seitz
Slide 80
Subsampling with Gaussian pre-filtering G 1/4G 1/8Gaussian 1/2
Slide by Steve Seitz
Slide 81
Why does a lower resolution image still make sense to us? What
do we lose? Image:
http://www.flickr.com/photos/igorms/136916757/http://www.flickr.com/photos/igorms/136916757/
Slide 82
Why do we get different, distance-dependent interpretations of
hybrid images? ?
Slide 83
Salvador Dali invented Hybrid Images? Salvador Dali Gala
Contemplating the Mediterranean Sea, which at 30 meters becomes the
portrait of Abraham Lincoln, 1976
Slide 84
Slide 85
Application: Hybrid Images A. Oliva, A. Torralba, P.G. Schyns,
Hybrid Images, SIGGRAPH 2006 Hybrid Images,
Slide 86
Early processing in humans filters for various orientations and
scales of frequency Perceptual cues in the mid-high frequencies
dominate perception When we see an image from far away, we are
effectively subsampling it Early Visual Processing: Multi-scale
edge and blob filters Clues from Human Perception
Slide 87
Hybrid Image in FFT Hybrid ImageLow-passed ImageHigh-passed
Image
Slide 88
Why do we get different, distance-dependent interpretations of
hybrid images? ? Perception
Slide 89
Things to Remember Sometimes it makes sense to think of images
and filtering in the frequency domain Fourier analysis Can be
faster to filter using FFT for large images (N logN vs. N 2 for
auto- correlation) Images are mostly smooth Basis for compression
Remember to low-pass before sampling
Slide 90
Sharpening revisited What does blurring take away? original
smoothed (5x5) detail = sharpened = Lets add it back:
originaldetail +