ME/CS 132:Introduction to

Vision-based Robot Navigation!Low-level Image Processing"

Larry Matthies"lhm@jpl.nasa.gov, 818-354-3722"

Announcements"

4-Jan-2011 lhm - 2 ME/CS 132

•  First homework grading is done!•  Second homework is due next Tuesday!•  Third homework will be due Feb 8; combines material on

outlier detection and egomotion!•  (Next quarter)!

Recap and Where Weʼre Going:1st Module (Done)"

4-Jan-2011 lhm - 3 ME/CS 132

•  Image formation, cameras, and camera calibration!–  Illumination and radiometry!

•  Sources of light – sunlight, thermal emission, night sky glow!•  Propagation of light – reflection from surfaces, attenuation in media!

–  Cameras!•  Basic optics!•  Camera architectures!•  Image detectors – materials, architectures, performance, for various

regions of the EM spectrum!–  Geometric camera modeling and calibration!

Recap and Where Weʼre Going:2nd Module (Midway Through)"

4-Jan-2011 lhm - 4 ME/CS 132

•  Visual motion estimation and 3-D perception!–  Low-level image processing!–  Feature detection, matching, and outlier detection!–  Pose estimation (egomotion) and visual odometry!–  Dense range imaging with stereo vision!–  Other range sensors, range data analysis, introduction to robots!

•  1-week mini-project on visual localization using a stereo camera head to match landmark points!

?

Recap and Where Weʼre Going:3rd Module"

4-Jan-2011 lhm - 5 ME/CS 132

•  State estimation, localization, and mapping!–  Introduction to estimation!–  Linear Kalman filter!–  Extended Kalman filter!–  Particle filters and the UKF!–  Simultaneous localization and mapping!

•  1-week mini-project on stereo-based SLAM for localization and occupancy grid mapping with ladar!

This Lecture"

4-Jan-2011 lhm - 6 ME/CS 132

•  Point operators!•  Neighborhood operators!•  Edge detection!•  Image pyramids!•  Geometric image transformations!•  Reading material:!

–  Today: Szeliski ch. 3 and sections 4.2-4.3!–  Next Tuesday: Szeliski ch. 11!

Images as Functions"

4-Jan-2011 lhm - 7 ME/CS 132

Interpret image either as:!•  Continuous image f(x,y)!•  Discrete array f[x,y]!Images are affected by:!•  Noise!•  Nonlinear distortions of

intensity and geometry!

x

yf(x,y)

Image Noise"

4-Jan-2011 lhm - 8 ME/CS 132

Image Distortions"

4-Jan-2011 lhm - 9 ME/CS 132

Point Operators"

4-Jan-2011 lhm - 10 ME/CS 132

•  Output pixel = function of one input pixel in one or more images!

•  Examples!–  Monadic linear: bias and gain adjustment, !

–  Monadic nonlinear: gamma correction, !

–  Dyadic linear: image blending, !

g(x) = h(f(x)) or g(x) = h(f0(x), ... , fn(x))

g(i, j) = h(f(i, j))

g(x) = a f(x) + b

g(x) = [f(x)]1/ϒ

g(x) = (1 - α) f0(x) + α f1(x)

Point Operators:Histogram Equalization"

4-Jan-2011 lhm - 11 ME/CS 132

g(x) = h(f(x)) or g(x) = h(f0(x), ... , fn(x))

g(i, j) = h(f(i, j))

g(x) = a f(x) + b

g(x) = [f(x)]1/ϒ

g(x) = (1 - α) f0(x) + α f1(x)

Point Operators:Color Space Conversion (e.g. RGB to HSV)"

4-Jan-2011 lhm - 12 ME/CS 132

HSV(i, j) = f ( RGB(i, j) )

Application of Color Space Conversion:Simple Segmentation"

4-Jan-2011 lhm - 13 ME/CS 132

(a) (b)

(c) (d)

Neighborhood Operators (or Window Operators, or Local Operators)"

4-Jan-2011 lhm - 14 ME/CS 132

•  Slide one small window of numbers over the image and compute some function, replacing the central pixel in the image with the output of the function.!

Linear Filters"

4-Jan-2011 lhm - 15 ME/CS 132

•  h() is called the filter, or kernel, or mask; entries in h() are the filter coefficients. Abbreviated notation:!

Correlation!

Convolution!

•  Can also be written:!

Some Basic Properties of Convolution"

4-Jan-2011 lhm - 16 ME/CS 132

•  Commutative:!

•  Associative:!

•  Distributes over addition:!

•  Differentiation:!

•  Shift invariant!

f * g = g * f!

(f * g)ʼ = fʼ * g = f * gʼ!

f * (g * h) = (f * g) * h!

f * (g + h) = f * g + f * h!

Neighborhood Operators:Border Effects"

4-Jan-2011 lhm - 17 ME/CS 132

•  When applying convolution with a KxK kernel, the result is undefined for pixels closer than K pixels to the border of the image!

•  Options: !

K

Warp around!Expand/Pad!

Crop!

1/9 1/9 1/9

1/9 1/9 1/9

1/9 1/9 1/9

Kernel:

0.04

0.04

0.04

0.04

0.04

0.04

0.04

0.04

0.04

0.04

0.04

0.04

0.04

0.04

0.04

0.04

0.04

0.04

0.04

0.04

0.04

0.04

0.04

0.04

0.04

Kernel:

Kernel: 15 x 15

matrix of value 1/225

1-D:

2-D:

Slight abuse of notations: We ignore the normalization constant such that

, σ = 5

Kernel:

Simple Averaging

Gaussian Smoothing

Image Noise

Gaussian Smoothing to Remove Noise

σ = 2 σ = 4 No smoothing

Computational Issues for Filters:Separability and Moving Averages"

4-Jan-2011 lhm - 27 ME/CS 132

•  When a KxK filter is equivalent to a Kx1 and a 1xK filter!

•  Reduces number of multiplies from K2 to 2K!

•  For box filter, moving average makes cost independent of K!

Some Other (Nonlinear) Neighborhood Operators:Median Filter"

4-Jan-2011 lhm - 28 ME/CS 132

•  Replace center pixel of KxK window with the mean value of all pixels in the window!

•  Good for removing large noise spikes without blurring image!

Original image! Gaussian filtered! Median filtered!

Some Other (Nonlinear) Neighborhood Operators:Bilateral Filter for Edge-Preserving Smoothing"

4-Jan-2011 lhm - 29 ME/CS 132

Some Other (Nonlinear) Neighborhood Operators:Bilateral Filter for Edge-Preserving Smoothing"

4-Jan-2011 lhm - 30 ME/CS 132

Original image! 15x15 box filter! 15x15 bilateral filter!

Other Useful Neighborhood Operators:Feature Detectors (Recall Earlier Lecture)"

4-Jan-2011 lhm - 31 ME/CS 132

(Implementation note: moving

averages)!

Other Useful Neighborhood Operators:Image Matching (Recall Earlier Lecture)"

4-Jan-2011 lhm - 32 ME/CS 132

•  Sum squared difference (SSD) vs. sum absolute difference (SAD)!

•  Normalization: why? Many approaches.!

Other Useful Neighborhood Operators"

4-Jan-2011 lhm - 33 ME/CS 132

•  Morphology!

•  Distance transform!

•  Connected components!

Edge Detection: What is an Edge?"

4-Jan-2011 lhm - 34 ME/CS 132

•  Local maxima of rate of change of intensity!

Origin of Edges"

4-Jan-2011 lhm - 35 ME/CS 132

•  Many factors!•  Sometimes care which factor applies; sometimes can

determine that!

depth discontinuity

surface color discontinuity

illumination discontinuity

surface normal discontinuity

Image Derivatives:1-D Case"

4-Jan-2011 lhm - 36 ME/CS 132

•  We want to compute, at each pixel (x,y) the derivatives:!•  In the discrete case we could take the difference

between the left and right pixels:!

•  Convolution of the image by !

•  Problem: Increases noise!

-1 0 1

Difference between Actual image values

True difference (derivative)

Twice the amount of noise as in the original image

Edges in 1-D:Effects of Noise"

4-Jan-2011 lhm - 37 ME/CS 132

Where is the edge?!

Solution: Smooth First"

4-Jan-2011 lhm - 38 ME/CS 132 Where is the edge? Look for peaks in !

€

f (x)

€

h(x)

€

h ∗ f

€

∂∂x(h ∗ f )

€

∂∂x(h ∗ f )

Derivative Property of Convolution:Saves One Step"

4-Jan-2011 lhm - 39 ME/CS 132

€

f (x)

€

∂∂xh

€

∂∂x(h ∗ f ) =

∂∂xh

⎛

⎝ ⎜

⎞

⎠ ⎟ ∗ f

€

∂∂xh

⎛

⎝ ⎜

⎞

⎠ ⎟ ∗ f

Laplacian of Gaussian"

4-Jan-2011 lhm - 40 ME/CS 132 Where is the edge? Zero crossing of bottom graph!

€

f (x)

€

∂ 2

∂x 2h

€

∂ 2

∂x 2h

⎛

⎝ ⎜

⎞

⎠ ⎟ ∗ f

2-D Edge Detection Filters"

is the Laplacian operator:

Laplacian of Gaussian

Gaussian derivative of Gaussian

4-Jan-2011 lhm - 41 ME/CS 132

DOG Approximation to LOG"

4-Jan-2011 lhm - 42 ME/CS 132

The Effect of Scale on Edge Detection"

4-Jan-2011 lhm - 43 ME/CS 132

larger

Scale space (Witkin 83)

€

σ

larger

€

σ

θ

Edge pixels are at local maxima of gradient magnitude Gradient computed by convolution with Gaussian derivatives Gradient direction is always perpendicular to edge direction

I

Applying the Gradient Magnitude Operator"

4-Jan-2011 lhm - 45 ME/CS 132

4-Jan-2011 lhm - 46 ME/CS 132

Different Thresholds Applied"to the Gradient Magnitude"

Additional steps:!

•  Thresholding with hysteresis!

•  Thinning (non-maximum suppression)!

•  Linking!

Sampling an Image"

4-Jan-2011 lhm - 47 ME/CS 132

Examples of GOOD sampling!

Undersampling and Aliasing"

4-Jan-2011 lhm - 48 ME/CS 132

Examples of BAD sampling -> Aliasing!

Downsampling and Aliasing"

4-Jan-2011 lhm - 49 ME/CS 132

Sample every other pixel to go left to right!

Downsampling with Smoothing(Gaussian, 1 Sigma)"

4-Jan-2011 lhm - 50 ME/CS 132

Sample every other pixel to go left to right!

Downsampling with Smoothing(Gaussian, 1.4 Sigma)"

4-Jan-2011 lhm - 51 ME/CS 132

Sample every other pixel to go left to right!

How do you know how much to smooth, with what kind of filter? Fourier transform theory.!

Intuitive Introduction to Fourier Transforms"

4-Jan-2011 lhm - 52 ME/CS 132

Intuition for Two Dimensions"

4-Jan-2011 lhm - 53 ME/CS 132

“dot” =

A measure of image content at this frequency and orientation

Multiresolution Image Representations"

Known as a Gaussian Pyramid [Burt and Adelson, 1983]"

Gaussian Pyramids have all sorts of applications in computer vision!

4-Jan-2011 lhm - 54 ME/CS 132

Gaussian Pyramid Construction"

4-Jan-2011 lhm - 55 ME/CS 132

filter kernel

Repeat!•  Filter!•  Subsample!

Until minimum resolution reached !•  can specify desired number of levels (e.g., 3-level pyramid)!

The whole pyramid is only 4/3 the size of the original image!

Computer Vision - A Modern Approach Set: Pyramids and Texture

Slides by D.A. Forsyth

Laplacian Pyramid Construction"

4-Jan-2011 lhm - 57 ME/CS 132

•  Given input I •  Construct Gaussian pyramid IG

1,..,IGn

•  Take the difference between consecutive levels:!•  IL

k = IGk – IG

k-1

•  Image ILk is an approximation of the Laplacian at

scale number k –  Laplacian is a band-pass filter: Both high frequencies

(edges and noise) and low frequencies (slow variations of intensity across the image)!

Computer Vision - A Modern Approach Set: Pyramids and Texture

Slides by D.A. Forsyth

Another Example"

4-Jan-2011 lhm - 59 ME/CS 132

Geometric Image Transformations (e.g. Barrel Distortion Correction, Rectification)"

4-Jan-2011 lhm - 60 ME/CS 132

Geometric Image Transformations (e.g. Barrel Distortion Correction, Rectification)"

4-Jan-2011 lhm - 61 ME/CS 132