Embed Size (px)
Citation preview
CS223, Jana Kosecka
Images
- photometric aspects of image formation- gray level images- linear/nonlinear filtering- edge detection- corner detection
CS223, Jana Kosecka
Image Brightness values
I(x,y)
CS223, Jana Kosecka
Image model
Analog intensity function
Temporal/spatial sampled function
Quantization of the gray levels
Point sets
Random fields
List of image features, regions
Mathematical tools
Analysis
Linear algebra
Numerical methods
Set theory, morphology
Stochastic methods
Geometry, AI, logic
CS223, Jana Kosecka
Basic Photometry
Radiometric model of image formation
CS223, Jana Kosecka
Basic ingredients
Radiance – amount of energy emitted along certain direction
Iradiance – amount of energy received along certain direction
BRDF – bidirectional reflectance distribution – portion of the energy coming from direction reflected to direction
Lambertian surfaces – the appearance depends only on radiance, not on the viewing direction
Image intensity for a Lambertian surface
CS223, Jana Kosecka
Images
• Images contain noise – sources sensor quality, light fluctuations, quantization effects
CS223, Jana Kosecka
Image Noise Models
• Additive noise: most commonly used
• Multiplicative noise:
• Impulsive noise (salt and pepper):
lxiiyi
lxjiIjiI
)(
if),(ˆ),(
minmaxmin
• Noise models: gaussian, uniform
• Noise Amount: SNR = s/ n
CS223, Jana Kosecka
Image filtering
• How can we reduce the noise in the image
• Image acquisition noise due to light fluctuations and sensor noise can be reduced by acquiring a sequence of images and averaging them
• Computation of simple features
• First stage of visual processing
CS223, Jana Kosecka
Image Processing
1D signal and its sampled version
f = { f(1), f(2), f(3), …, f(n)}
f = {0, 1, 2, 3, 4, 5, 5, 6, 10 }
CS223, Jana Kosecka
Discrete time system
• Maps 1 discrete time signal to another
hf[x] g[x]
• Special class of systems – linear , time-invariant systems
Superposition principle
Shift (time) invariant – shift in input causes shift in output
CS223, Jana Kosecka
The output the linear system is related to the input and the transferfunction via convolution
Convolution sum:
unit impluse – if x = 0 it’s 1 and zero everywhere else
Every discrete time signal can be written as a sum of scaled and shifted impulses
Convolution sum:
Notation:
hf g
filterf g
CS223, Jana Kosecka
Averaging filterOriginal image
Smoothed image
CS223, Jana Kosecka
Averaging filter
Box filter
Averaging filter center pixel weighted more
and 0 everywhere else
Ex. cont.
CS223, Jana Kosecka
Convolution in 2D
1010 1111 1010
99 1010 1111
1010 99 1010
11
10101010
22
9900
99
00
9999
9999
00
11
9999
1010
1010 1111
110011
11111111
1111
1111
10101010
ff
111111
11
11 11
1111
11
hh
XX XX XX
XX 1010
XX
XX
XXXX
XX
XX
XX
XX
XX
XXXX
XX
XX
XX
XXXX
1/91/9
gg
1/9.(10x1 + 11x1 + 10x1 + 9x1 + 10x1 + 11x1 + 10x1 + 9x1 + 10x1) = 1/9.(10x1 + 11x1 + 10x1 + 9x1 + 10x1 + 11x1 + 10x1 + 9x1 + 10x1) = 1/9.( 90) = 101/9.( 90) = 10
CS223, Jana Kosecka
Example:
1/9.(10x1 + 0x1 + 0x1 + 11x1 + 1x1 + 0x1 + 10x1 + 0x1 + 2x1) = 1/9.(10x1 + 0x1 + 0x1 + 11x1 + 1x1 + 0x1 + 10x1 + 0x1 + 2x1) = 1/9.( 34) = 3.77781/9.( 34) = 3.7778
1010 1111 1010
99 1010 1111
1010 99 1010
11
10101010
22
9900
99
00
9999
9999
00
11
9999
1010
1010 1111
110011
11111111
1111
1111
10101010
II
111111
11
11 11
1111
11
FF
XX XX XX
XX 1010
XX
XX
XXXX
XX
XX
XX
XX
XX
XXXX
XX
XX
XX
XXXX
1/91/9
OO4477 11
CS223, Jana Kosecka
Example:
1/9.(10x1 + 9x1 + 11x1 + 9x1 + 99x1 + 11x1 + 11x1 + 10x1 + 10x1) = 1/9.(10x1 + 9x1 + 11x1 + 9x1 + 99x1 + 11x1 + 11x1 + 10x1 + 10x1) = 1/9.( 180) = 201/9.( 180) = 20
1010 1111 1010
99 1010 1111
1010 99 1010
11
10101010
22
9900
99
00
9999
9999
00
11
9999
1010
1010 1111
110011
11111111
1111
1111
10101010
II
11
1111
11
11 111111
11
FF
XX XX XX
XX 1010
XX
XX
XXXX
XX
XX
XX
XX
XX
XXXX
XX
XX
XX
XXXX
1/91/9
OO4477 11
2020
CS223, Jana Kosecka
How big should the mask be?
• The bigger the mask,– more neighbors contribute– bigger noise spread.– more blurring.– more expensive to compute
• Signal frequencies shared with noise are lost • Impulsive noise is diffused but not removed
Limitations of averaging
CS223, Jana Kosecka
Gaussian Filter
• Better option for blurring– The coefficients are samples of a 1D Gaussian.– Gives more weight at the central pixel and less weights
to the neighbors.– The further away the neighbors, the smaller the weight.
– Gaussian filterSamples from the continuous Gaussian
• Gaussian filter is the only one that has the same shape in the space and frequency domains.
• There are no secondary lobes – i.e. a truly low-pass filter
CS223, Jana Kosecka
How big should the mask be?
• The std. dev of the Gaussian determines the amount of smoothing.
• The samples should adequately represent a Gaussian • For a 98.76% of the area, we need
m = 55.(1/) 2 0.796, m 5
g[x] = [0.136, 0.6065, 1.00, 0.606, 0.136]
5-tap filter
CS223, Jana Kosecka
Image Smoothing
• Convolution with a 2D Gaussian filter
• Gaussian filter is separable, convolution can be accomplished as two 1-D convolutions
CS223, Jana Kosecka
Non-linear Filtering
• Replace each pixel with the MEDIAN value of all the pixels in the neighborhood
• Non-linear• Does not spread the noise• Can remove spike noise• Expensive to run
CS223, Jana Kosecka
Example:
10,11,10,9,10,11,10,9,1010,11,10,9,10,11,10,9,10
1010 1111 1010
99 1010 1111
1010 99 1010
11
10101010
22
9900
99
00
9999
9999
00
11
9999
1010
1010 1111
110011
11111111
1111
1111
10101010
II
XX XX XX
XX 1010
XX
XX
XXXX
XX
XX
XX
XX
XX
XXXX
XX
XX
XX
XXXX
OO
9,9,10,10,10,10,10,11,119,9,10,10,10,10,10,11,11sortsort
medianmedian
CS223, Jana Kosecka
Example:
11,10,0,10,11,1,9,10,011,10,0,10,11,1,9,10,0
1010 1111 1010
99 1010 1111
1010 99 1010
11
10101010
22
9900
99
00
9999
9999
00
11
9999
1010
1010 1111
110011
11111111
1111
1111
10101010
II
XX XX XX
XX 1010
XX
XX
XXXX
XX
XX
XX
XX
XX
XXXX
XX
XX
XX
XXXX
OO
0,0,1,9,10,10,10,11,110,0,1,9,10,10,10,11,11sortsort
medianmedian
1010 11 11
CS223, Jana Kosecka
Example:
10,9,11,9,99,11,11,10,1010,9,11,9,99,11,11,10,10
1010 1111 1010
99 1010 1111
1010 99 1010
11
10101010
22
9900
99
00
9999
9999
00
11
9999
1010
1010 1111
110011
11111111
1111
1111
10101010
II
XX XX XX
XX 1010
XX
XX
XXXX
XX
XX
XX
XX
XX
XXXX
XX
XX
XX
XXXX
OO
9,9,10,10,10,11,11,11,999,9,10,10,10,11,11,11,99sortsort
medianmedian
1010
99
CS223, Jana Kosecka
Image Features
Local, meaningful, detectable parts of the image.We will look at edges and corners
CS223, Jana Kosecka
Image Features – Edges, Corners
• Look for detectable, meaningful parts of the image• Edges are detected at places where the image values
exhibit sharp variation
Gray valueGray value
columncolumncolumncolumn
EdgeEdge EdgeEdge
CS223, Jana Kosecka
Edge detection (1D)
F(x)F(x)
xxF ’(x)F ’(x)
xx
Edge= sharp variationEdge= sharp variation
Large first derivativeLarge first derivative
CS223, Jana Kosecka
Digital Approximation of 1st derivatives
x
xfxxf
dx
xdfx
)()(lim
)(0
2
)1()1()(
xfxf
dx
xdf
XX
f(x)f(x)
X-1X-1 X+1X+1
-1-1 00 11Convolve with:Convolve with:
CS223, Jana Kosecka
Edge Detection (2D)
-1-1 00 11Convolve with:Convolve with:
Vertical Edges:Vertical Edges:
Horizontal Edges:Horizontal Edges:
Convolve with:Convolve with:
-1-1
00
11
CS223, Jana Kosecka
Noise cleaning and Edge Detection
NoiseNoiseFilterFilter
EdgeEdgeDetectionDetection
I(x,y)I(x,y)
We need to also deal with noiseWe need to also deal with noise-> Combine Linear Filters-> Combine Linear Filters
E(x,y)E(x,y)
CS223, Jana Kosecka
Noise Smoothing & Edge Detection
Convolve with:Convolve with: -1-1 00 11
-1-1 00 11
-1-1 00 11
Vertical Edge DetectionVertical Edge Detection
Nois
e S
mooth
ing
Nois
e S
mooth
ing
This mask is called the (vertical) Prewitt Edge DetectorThis mask is called the (vertical) Prewitt Edge Detector
Outer product of box filter [1 1 1]Outer product of box filter [1 1 1]TT and [-1 0 1] and [-1 0 1]
CS223, Jana Kosecka
Noise Smoothing & Edge Detection
Convolve with:Convolve with: -1-1 -1-1 -1-1
00 00 00
11 11 11
Noise SmoothingNoise Smoothing
Hori
zonta
l Edg
e D
ete
ctio
nH
ori
zonta
l Ed
ge D
ete
ctio
n
This mask is called the (horizontal) Prewitt Edge DetectorThis mask is called the (horizontal) Prewitt Edge Detector
CS223, Jana Kosecka
Sobel Edge Detector
Convolve with: -1-1 00 11
-2-2 00 22
-1-1 00 11
and -1-1 -2-2 -1-1
00 00 00
11 22 11
Gives more weightto the center pixels
CS223, Jana Kosecka
Example
-1-1 00 11
-1-1 00 11
-1-1 00 11
-1-1 -1-1 -1-1
00 00 00
11 11 11
00 00000000
00 00000000
00
0000
00
0000
5050 5050 5050
5050 5050 5050
5050 5050 5050
5050 5050 00
100100
150150 150150
00
00
100100
00
00
00
00
00
00
00 00 00 00 00
00 00 00150150 150150
00 00 00 00 00
00
00
00
00 00 00 00 0000
5050
5050
00 00 00
100100
100100
150150
150150
150150
150150xI
yI
CS223, Jana Kosecka
Image Derivatives
We know better alternative to smoothing Smooth using Gaussian filter
Taking a derivative – linear operation (take the derivative of the filter)
g(x) is a 1-D Gaussian filter, g(x,y) – 2-D Gaussian filter
CS223, Jana Kosecka
Gaussian and its derivative
CS223, Jana Kosecka
Vertical edges First derivative
CS223, Jana Kosecka
Horizontal edges
• Image Gradient
Gradient Magnitude
CS223, Jana Kosecka
Gradient Orientation
CS223, Jana Kosecka
Orientation histogram
CS223, Jana Kosecka
• A point on a line is hard to match.
Corner detection
Intuition:
• Right at corner, gradient is ill defined.
• Near corner, gradient has two different values.
CS223, Jana Kosecka
Formula for Finding Corners
2
2
yyx
yxx
III
IIIC
We look at matrix:
Sum over a small region, the hypothetical corner
Gradient with respect to x, times gradient with respect to y
Matrix is symmetric
CS223, Jana Kosecka
2
12
2
0
0
yyx
yxx
III
IIIC
First, consider case where:
This means all gradients in neighborhood are:
(k,0) or (0, c) or (0, 0) (or off-diagonals cancel).
What is region like if:
1. ?
2. ?
3. and ?
4. and ?
CS223, Jana Kosecka
General Case:
In general case from linear algebra, it follows that because C is symmetric:
RRC
2
11
0
0
With R being a rotation matrix. So every case can be intrepreted like one on last slide.
CS223, Jana Kosecka
Corner Detection
• Filter image.• Compute magnitude of the gradient everywhere.• We construct C in a window.• Compute eigenvalues and • If they are both big, we have a corner• Or if smalest eigenvalue of C is bigger than - mark
pixel as candidate feature point
CS223, Jana Kosecka
• Alternatively feature quality function (Harris Corner Detector)
Point Feature ExtractionHarris Corner Detector
CS223, Jana Kosecka
Harris Corner Detector - Example
CS223, Jana Kosecka
• Compute image derivatives • if gradient magnitude > and the value is a local max. along gradient direction – pixel is an edge candidate• how to detect one pixel thin edges ?
gradient magnitudeoriginal image
Edge Detection
CS223, Jana Kosecka
• The magnitude image Es has the magnitudes of the smoothed gradient.
• Sigma determines the amount of smoothing.
• Es has large values at edges:
– Find local maxima
Canny Edge Detector
ThTh
CS223, Jana Kosecka
Nonmaximum supression
• The inputs are Es & Eo Magnitude and orientation• Consider 4 directions D={ 0,45,90,135} wrt x
• For each pixel (i,j) do:1. Find the direction dD s.t. d Eo(i,j) (normal to the edge)
2. If {Es(i,j) is smaller than at least one of its neigh. along d} • IN(i,j)=0
• Otherwise, IN(i,j)= Es(i,j)
• The output is the thinned edge image INxx xx
CS223, Jana Kosecka
Thresholding
• Edges are found by thresholding the output of NONMAX_SUPRESSION
• If the threshold is too high:– Very few (none) edges
• High MISDETECTIONS, many gaps• If the threshold is too low:
– Too many (all pixels) edges• High FALSE POSITIVES, many extra edges
CS223, Jana Kosecka
Hysteresis Thresholding
Es(i,j)> Es(i,j)> HH
Es(i,j)<HEs(i,j)<HEs(i,j)>LEs(i,j)>L
Es(i,j)<LEs(i,j)<LEs(i,j)>LEs(i,j)>L
Strong edges reinforce adjacent weak edges
CS223, Jana Kosecka
Finding lines in an image
• Option 1:– Search for the line at every possible
position/orientation– What is the cost of this operation?
• Option 2:– Use a voting scheme: Hough transform
CS223, Jana Kosecka
Other edge detectors - second-order derivative filters (1D)
• Peaks of the first-derivative of the input signal, correspond to “zero-crossings” of the second-derivative of the input signal.
F(x)F(x) F ’(x)F ’(x)
xx
F’’(x)F’’(x)
CS223, Jana Kosecka
Edge Detection (2D)
1D1D 2D2D
I(x)I(x) I(x,y)I(x,y)
dd22I(x)I(x)dxdx22
= 0= 0
xx
yy
||I(x,y)| =(II(x,y)| =(Ix x 22(x,y) + I(x,y) + Iyy
22(x,y))(x,y))1/2 1/2 > Th> Th
tan tan = I = Ixx(x,y)/ I(x,y)/ Iyy(x,y) (x,y)
F(x)F(x)
xx
dI(x)dI(x)dxdx
> Th> Th
22I(x,y) =II(x,y) =Ix x x x (x,y) + I(x,y) + Iyy yy (x,y)=0(x,y)=0
LaplacianLaplacian
CS223, Jana Kosecka
Notes about the Laplacian:
• 22I(x,y) is a SCALARI(x,y) is a SCALAR Can be found using a SINGLE maskCan be found using a SINGLE mask Orientation information is lostOrientation information is lost
22I(x,y) is the sum of SECOND-order derivativesI(x,y) is the sum of SECOND-order derivatives– But taking derivatives increases noiseBut taking derivatives increases noise– Very noise sensitive!Very noise sensitive!
• It is always combined with a smoothing operation:It is always combined with a smoothing operation:• Filter Laplacian of Gaussian LOG filterFilter Laplacian of Gaussian LOG filter
SmoothSmooth LaplacianLaplacianI(x,y)I(x,y) O(x,y)O(x,y)
CS223, Jana Kosecka
1D Gaussian2
2
2)( x
exg
2
2
2
2
22
22
22
1)('
xx
ex
xexg
2D2D
““Mexican Hat”Mexican Hat”
2
2
23
2
)1
()(''
x
ex
xg
CS223, Jana Kosecka
Derivative of Gaussian Filters
Measure the image gradient and its direction at Measure the image gradient and its direction at different scales (use a pyramid).different scales (use a pyramid).
CS223, Jana Kosecka
The Laplacian Pyramid
• Building a Laplacian pyramid:Building a Laplacian pyramid:– Create a Gaussian pyramidCreate a Gaussian pyramid– Take the difference between one Gaussian pyramid Take the difference between one Gaussian pyramid
level and the next (before subsampling)level and the next (before subsampling)• PropertiesProperties
– Also known as the difference-of-Gaussian function, Also known as the difference-of-Gaussian function, which is a close approximation to the Laplacianwhich is a close approximation to the Laplacian
– It is a band pass filter - each level represents a It is a band pass filter - each level represents a different band of spatial frequencies different band of spatial frequencies
• Reconstructing the original image:Reconstructing the original image:– Reconstruct the Gaussian pyramid starting at top Reconstruct the Gaussian pyramid starting at top
layerlayer
CS223, Jana Kosecka
Gaussianpyramid
CS223, Jana Kosecka
LaplacianPyramid(note top imageis from Gaussian)
CS223, Jana Kosecka
Add more oriented filters(Malik & Perona, 1990)
CS223, Jana Kosecka
Gabor filters: Product of a Gabor filters: Product of a Gaussian with sine or cosineGaussian with sine or cosine
Top row shows anti-symmetric Top row shows anti-symmetric (or odd) filters, bottom row the(or odd) filters, bottom row thesymmetric (or even) filters.symmetric (or even) filters.
No obvious advantage to any one No obvious advantage to any one type of oriented filters.type of oriented filters.
Alternative: Gabor filters
CS223, Jana Kosecka
An edge is not a line...
• How can we detect lines ?
