!"#$"#%&

#&

Image Enhancement in the Spatial Domain

Jesus J. Caban

Outline

1.  Final Project / Paper Presentation

2.  Questions about homework / OpenCV?

3.  Review topics from last class

4.  Finish talking about spatial image enhancement

5.  Start Fourier Analysis

!"#$"#%&

'&

Paper Presentation !  Presenters:

!  Prepare a 30 – 35 mins talk about your topic !  Try to “connect” the two papers !  If needed, read other papers !  Undergraduates: 20 – 30mins !  Evaluation Form

!  Rest of the class: !  Read each of the papers !  Prepare and submit a thoughtful question about the

papers

Graduate students will be given 10 additional minutes (30 – 35 mins in total) for their paper presentation. The additional time should be used to better cover some of the previous work as well as explain the formulation of the paper.

Recall:

Homework / OpenCV

1.  Download and install cmake from cmake.org (default location)

2.  Download OpenCV

3.  unzip OpenCV-2.1.0-win.zip && cd OpenCV-2.1.0

4.  cmake .

5.  Make

6.  sudo make install

7.  Add theses two lines to my .bash_profile

export PYTHONPATH=$PYTHONPATH:/usr/local/lib/python2.6/site-packages

export DYLD_LIBRARY_PATH=/usr/local/lib OR

export LD_LIBRARY_PATH=/usr/local/lib

8.  Open a new terminal

9.  cd OpenCV-2.1.0/samples/python/

10.  python edge.py

!"#$"#%&

(&

OpenCV ##Load / Create

src_image = cv.LoadImage(“image.jpg”, cv.CV_LOAD_IMAGE_GRAYSCALE) editted_image = cv.CreateImage(cv.GetSize(src_image), 8, 1);

##Processing / Filter cv.Smooth(src_image, editted_image,cv.CV_BLUR, 3,3,0)

#cv.Laplace(src_image, float_image, 3)

#cv.Merge(src_image, src_image, src_image, None, rgb_image)

##Show / Save

cv.SaveImage(“out.jpg”, editted_image) cv.NamedWindow("Result", 0)

cv.ShowImage( "Result", image) cv.WaitKey(0)

Make sure you understand the following examples: �

•  Fitellipse.py�•  Edge.py�•  Contours.py�•  Cv20squares.py�

Final Project !  Draft Proposal (10%)

!  Oct 4th

!  Revised Proposal (10%) !  Oct 25th

!  Literature survey (20%) !  Nov 3rd

!  Final paper (40%) !  Dec 6th

!  Final presentation (20%)

• When: Draft due Oct 4th

• What: One to two pages: 1.  Abstract 2.  Motivation 3.  Plan: What you plan to do 4.  How: Resources & APIs you plan to use 5.  Timeline

!"#$"#%&

)&

Need to talk to other students?

!  VANGOGH Lab ! Dr. Penny Rheingans and Dr. Marc Olano

!  Lab meetings: !  Wednesday, 1pm

!  ITE 365

Outline

1.  Final Project / Paper Presentation

2.  Questions about homework / OpenCV?

3.  Review topics from last class

4.  Finish talking about spatial image enhancement

5.  Start Fourier Analysis

!"#$"#%&

$&

Point Processing !  Based on the intensity of a single pixel only as opposed to a

neighborhood or region

Histogram Equalization

!"#$"#%&

*&

Histogram Equalization 1) Probability of Occurrence

2) Create Cumulative Distribution Function

3) Map old value to new value

Histogram Equalization

!"#$"#%&

+&

Histogram Matching

!  The histogram processing methods discussed previously are global, in the sense that pixels are modified by a transformation function based on the gray-level content of an entire image.

!  However, there are cases in which it is necessary to enhance details over small areas in an image.

original

Local Enhancement

global Local (7x7)

!"#$"#%&

,&

!  Moments can be determined directly from a histogram much faster than they can from the pixels directly.

!  Let r denote a discrete random variable representing discrete gray-levels in the range [0,L-1], and p(ri) denote the normalized histogram component corresponding to the ith value of r, then the nth moment of r about its mean is defined as

where m is the mean value of r

!  For example, the second moment (also the variance of r) is

Use of Histogram Statistics for Image Enhancement

!  Two uses of the mean and variance for enhancement purposes: ! The global mean and variance (global means for the entire image) are

useful for adjusting overall contrast and intensity.

! The mean and standard deviation for a local region are useful for correcting for large-scale changes in intensity and contrast

Use of Histogram Statistics for Image Enhancement

!"#$"#%&

!&

Enhancement based on local statistics

Image Operations

!  Just as with numbers, we can apply different operations to an image ! Addition: + !  Subtraction: - ! Multiplication: * ! And: && ! Or: || ! Derivative:

! Average ! Mean

!

dxdy

!"#$"#%&

#%&

!  In general, linear filtering of an image f of size MxN is given by

!  This concept called convolution. Filter masks are sometimes called convolution masks or convolution kernels.

Basics of Spatial Filtering

Smoothing

1/9 1/9 1/9

1/9 1/9

1/9

1/9 1/9

1/9

1/16 2/16 1/16

2/16 4/16

2/16

1/16 2/16

1/16

!"#$"#%&

##&

1st Order Derivative The formula for the 1st derivative of a function is as follows:

It’s just the difference between subsequent values and measures the rate of change of the function

2nd Order Derivative

The formula for the 2nd derivative of a function is as follows:

Simply takes into account the values both before and after the current value

!"#$"#%&

#'&

The Laplacian So, the Laplacian can be given as follows:

We can easily build a filter based on this

0 1 0

1 -4 1

0 1 0

Laplacian Image Enhancement

!"#$"#%&

#(&

Variants On The Simple Laplacian There are lots of slightly different versions of the Laplacian that can be used:

0 1 0

1 -4 1

0 1 0

1 1 1

1 -8 1

1 1 1

-1 -1 -1

-1 9 -1

-1 -1 -1

Simple Laplacian

Variant of Laplacian

1st Order Derivative

5 5 4 3 2 1 0 0 0 6 0 0 0 0 1 3 1 0 0 0 0 7 7 7 7

0 -1 -1 -1 -1 0 0 6 -6 0 0 0 1 2 -2 -1 0 0 0 7 0 0 0

!"#$"#%&

#)&

1st Derivative Filtering – The gradient Implementing 1st derivative filters is difficult in practice

For a function f(x, y) the gradient of f at coordinates (x, y) is given as the column vector:

1st Derivative Filtering The magnitude of this vector is given by:

For practical reasons this can be simplified as:

!

= Gx2 +Gy

2[ ]12

!

="f"x#

$ %

&

' ( 2

+"f"y#

$ %

&

' (

2)

* + +

,

- . .

12

!"#$"#%&

#$&

Gradient Estimation

1. Create orthogonal pair of filters,

!

h1(x,y)

!

h2(x,y)

!

"f (x,y) = f12(x,y) + f2

2(x,y)

3. Estimate the gradient magnitude:

2. Convolve image with each filter:

!

f1(x,y) = f (x,y)"h1(x,y)

!

f2(x,y) = f (x,y)"h2(x,y)

1st Derivative Filtering There is some debate as to how best to calculate these gradients:

which is based on these coordinates

z1 z2 z3

z4 z5 z6

z7 z8 z9

!"#$"#%&

#*&

Roberts Operator

!  Small kernel, relatively little computation !  First difference (diagonally) !  Very sensitive to noise !  Origin not at kernel center !  Somewhat anisotropic

!

h2(x,y) =1 00 "1#

$ %

&

' (

!

h1(x,y) =0 1"1 0#

$ %

&

' (

Noise

!  Noise is always a factor in images. !  Derivative operators are high-pass filters. !  High-pass filters boost noise!

!  Effects of noise on edge detection: !  False edges !  Errors in edge position

Key concept: Build filters to respond to edges and suppress noise.

!"#$"#%&

#+&

Prewitt Operator

!  Larger kernel, somewhat more computation !  Central difference, origin at center !  Smooths (averages) along edge, less sensitive to noise !  Somewhat anisotropic

•  3 x 3 kernel, same computation as Prewitt •  Central difference, origin at center •  Better smoothing along edge, even less sensitive to noise •  Still somewhat anisotropic

Sobel Operator

-1 -2 -1

0 0 0

1 2 1

-1 0 1

-2 0 2

-1 0 1

!"#$"#%&

#,&

Sobel Example

Sobel filters are typically used for edge detection

An image of a contact lens which is enhanced in order to make defects (at four and five o’clock in the image) more obvious

Roberts cross-gradient operators

Sobel operators

1st Derivative Filtering

!"#$"#%&

#!&

Example

Original Roberts

Roberts Prewitt

!"#$"#%&

'%&

Prewitt Sobel

1st & 2nd Derivatives Comparing the 1st and 2nd derivatives we can conclude the following:

!  1st order derivatives generally produce thicker edges !  2nd order derivatives have a stronger response to fine detail e.g.

thin lines !  1st order derivatives have stronger response to grey level step !  2nd order derivatives produce a double response at step changes

in grey level

!"#$"#%&

'#&

Effect of Noise

Where is the edge?

Multi-Filter Processing !  LoG: (Laplacian of Gaussian)

!  Because second derivative measurements on an image are very sensitive to noise. Images are often Gaussian smoothed before applying the Laplacian filter.

!  This pre-processing step reduces the high frequency noise components prior to the differentiation step.

!  DoG: (Difference of Gaussians) !  Given f(x,y), compute f’(x,y) using a Gaussian

blur !  Subtract images !  Threshold image

!"#$"#%&

''&

Canny edge detector

•  Based on the theoretical model that edges are corrupted by additive Gaussian noise

•  The first derivative of the Gaussian closely approximates the operator that optimizes the product of signal-to-noise ratio and localization (1986)

Canny edge detector

1.  Filter image with derivative of Gaussian

2.  Find magnitude and orientation of gradient

(Round orientations: 0o, 45o, 90o, 135o)

3.  Non-maximum suppression: !  Thin multi-pixel wide “ridges” down to single pixel width

!"#$"#%&

'(&

Non-maximum suppression At q, we have a maximum if the value is larger than those at both p and at r. Interpolate to get these values.

Non-maximum suppression

!  A(0o): edge if I(x,y) > I(x+1,y) && I(x-1,y)

!  A(90o): edge if I(x,y) > I(x,y+1) && I(x,y-1)

!  A(135o): edge if I(x,y) > I(x-1,y-1) && I(x+1,y+1)

!  A(45o): edge if I(x,y) > I(x+1, y-1) && I(x-1, y+1)

!"#$"#%&

')&

Canny edge detector

1.  Filter image with derivative of Gaussian

2.  Find magnitude and orientation of gradient

(Round orientations: 0o, 45o, 90o, 135o)

3.  Non-maximum suppression: !  Thin multi-pixel wide “ridges” down to single pixel width

4.  Linking of edge points

1.  Assume the marked point is an edge point.

2.  Construct the tangent to the edge curve (which is normal to the gradient at that point)

3.  Use tangent to predict the next points

Edge linking

!"#$"#%&

'$&

Example

Gradient magnitude

Canny

!"#$"#%&

'*&

Summary: Image Enhancement – Spatial Domain

!  Point Processing ! Contrast

!  Operations !  Convolution !  Derivatives !  Edge detection

Image Enhancement: Frequency Domain

!"#$"#%&

'+&

!  The spatial domain: !  The image plane !  For a digital image is a Cartesian coordinate system of discrete rows and columns.

At the intersection of each row and column is a pixel. Each pixel has a value, which we will call intensity.

!  The frequency domain : !  A (2-dimensional) discrete Fourier transform of the spatial domain

!  Enhancement : !  To “improve” the usefulness of an image by using some transformation on the

image. !  Often the improvement is to help make the image “better” looking, such as

increasing the intensity or contrast.

Recall: Image Enhancement - Spatial Domain

Jean Baptiste Joseph Fourier

! Fourier was born in Auxerre, France in 1768

! Published his work in 1822: “La Théorie Analitique de la Chaleur”

! His work was translated into English over 55+ years later (1878): “The Analytic Theory of Heat”

! Researchers didn’t pay too much attention to his work when it was first published

! Fourier Theory then became one of the most important mathematical theories in modern engineering

!"#$"#%&

',&

The General Idea

=

•  Any function that periodically repeats itself can be expressed as a sum of sines and cosines of different frequencies each multiplied by a different coefficient •  Resulting collection is called Fourier series

Fourier Series - Example

First 60 Fourier approximations

!"#$"#%&

'!&

Fourier Transform and Image Frequency

!  Determines frequency components that fit evenly across an image of input size L

!  The highest frequency that can be contained in a signal of length L is L/2

! At least two pixels are required to fully represent a cycle

Fourier Transform: Definition

!  !  Changes in u and v obtain different basis functions of different periods

(frequencies) !  Therefore, F(f(x,y)) is really a weighted sum of complex sinusoids of

varying frequency !  Point in the new image FT(u,v) represent the contribution of that

frequency to the original image

Sum of sine and cosine basis functions

!"#$"#%&

(%&

The Discrete Fourier Transform (DFT) 1.  Given an image f(x,y) or size MxN

x = 0, 1, 2…M-1 y = 0,1,2…N-1

2.  Discrete Fourier Transform of f(x, y) is given by

u = 0, 1, 2…M-1 v = 0, 1, 2…N-1 i= sqrt(-1)

N

M

Origin

Acknowledgements !  Some of the images and diagrams have been taken from the

Gonzalez et al, “Digital Image Processing” book.