10-1

((1100)) IImmaaggee SSeeggmmeennttaattiioonn - Image analysis

Low-level image processing: inputs and outputs are all images Mid-/High-level image processing: inputs are images; outputs are information or attributes of the images

Image analysis: extract useful information from the result of low-level image processing

Image segmentation: divides an image into meaningful parts or objects

- Image segmentation Two graylevel characteristics are generally considered: * Discontinuities: segment the image according to the abrupt change in

graylevels # E.g.: point, line, edge detection…

* Similarity: segment the image according to some predefined criteria of similarity

10-2

# E.g.: thresholding, region growing, region splitting and merging…

((aa)) DDeetteeccttiioonn ooff ddiissccoonnttiinnuuiittiieess - Point detection

By a mask operation Mask: Image: w1 w2 w3 z1 z2 z3 w4 w5 w6 z4 z5 z6 w7 w8 w9 z7 z8 z9

∑=

=→9

1

iii zwR

If |R| ≥ T, a point is detected isolated point (significant graylevel difference with neighboring pixels)

E.g., using the mask: −1 −1 −1 −1 8 −1 −1 −1 −1

* MATLAB: w = [-1 -1 -1; -1 8 -1; -1 -1 -1];

10-3

g = abs(imfilter(double(f), w) >= T; - Line detection

E.g., using the following masks: −1 −1 −1 −1 −1 2 2 −1 −1 −1 2 −12 2 2 −1 2 −1 −1 2 −1 −1 2 −1−1 −1 −1 2 −1 −1 −1 −1 2 −1 2 −1Horizontal (R1) 45° (R2) −45° (R3) Vertical (R4)

TRi ≥ : one-pixel wide line * MATLAB:

w = [-1 -1 -1; 2 2 2; -1 -1 -1]; g = abs(imfilter(double(f), w) >= T; % Horizontal line

- Edge detection Common method to detect discontinuities because isolated points or one-pixel wide lines are rare

Edge: the borders of regions with different graylevels Digital edge model:

10-4

Ramp edge * More clear edge: width of the ramp is narrow; fuzzy edge: width of the

ramp is wider * All pixels in the ramp are edge points * First derivatives

# Vector:

∂∂∂∂

=

=∇

yfxf

GG

y

xf

# Magnitude: [ ] yxyx GGGGf +≈+=∇=∇2/122)(mag f

Determines if a pixel is in the ramp edge

Perfect edge Ramp edge (more realistic in images)

10-5

# Direction of gradient:

−

GxGy1tan

* Second derivative: 2

2

2

22

yf

xff

∂∂

+∂∂

≈∇

Profile of graylevel edge

First derivative

Second derivativeZero-crossing

10-6

# Sign of second derivative determines if an edge point is on the bright or dark side - There are positive and negative values on a ramp edge - Zero-crossing: intersection of the 0-value line and the line connecting

the positive and negative points position of edge Definition: * Edge point: 2nd order derivative is greater than a threshold * Edge: set of connected edge points (need to predefine “connection”) * Edge segment: short edges Implementation: * 1st order derivative: gradient operator

# Sobel operators –1 –2 –1 –1 0 1 0 1 2 –2 –1 0 0 0 0 –2 0 2 –1 0 1 –1 0 1 1 2 1 –1 0 1 –2 –1 0 0 1 2

# Prewitt operators (including slanted line) –1 –1 –1 –1 0 1 0 1 1 –1 –1 0 0 0 0 –1 0 1 –1 0 1 –1 0 1 1 1 1 –1 0 1 –1 –1 0 0 1 1

10-7

# Roberts operators 1 0 0 1 0 –1 –1 0

* 2nd order derivative: Laplacian or LoG operator 0 −1 0 −1 −1 −1 0 0 −1 0 0 −1 4 −1 或 −1 8 −1 0 −1 −2 −1 0 0 −1 0 −1 −1 −1 −1 −2 16 −2 −1 0 −1 −2 −1 0 0 0 −1 0 0

Sobel

LoG

Thresholded LoG

Zero crossing

MATLAB bw = edge(im, method, parameters) * method: 'sobel', 'prewitt', 'roberts', 'log', 'zerocross', 'canny'…

10-8

((bb)) TThhrreesshhoollddiinngg - Thresholding: setting threshold T in histogram to segment the image

Threshold: T[x, y, p(x, y), f(x, y)] (x, y): pixel coordinate, p(x, y): region property, and f(x, y): graylevel

* Two-modal single thresholding

≤>

=TyxfTyxf

yxg),( if 1),( if 0

),( g(x, y) = 1: obj0; g(x, y) = 1: obj1 * Multimodal multilevel thresholding

T

Two-modal

T1 T2

Multimodal

10-9

<≤

<=M

21

1

),( if 1),( if 0

),( TyxfTyxfT

yxg

* T related to f(x, y) only: global thresholding * T related to f(x, y) and p(x, y) only: local thresholding * T related to f(x, y), p(x, y), and (x, y): dynamic (adaptive) thresholding E.g., Tiffany: T1, T2, T3, T4 = 150, 170, 190, 220

E.g., Tool: T = 127

10-10

- Global thresholding algorithm 1. Select an initial threshold T: usually, T = (graymax + graymin)/2 2. Segment the image using T: two groups of pixels G1 (< T) and G2 (≥ T) 3. Compute average intensities of G1 and G2: µ1 and µ2 4. Compute a new threshold T: T = (µ1 + µ2 )/2 5. Repeat Steps 2 to 4 until the difference in T in successive iterations is

smaller than a predefined parameter T0 MATLAB: T = 0.5*(double(min(f(:))) + double(max(f(:)))); % f: input image done = false; while ~done

10-11

g = f >= T; Tnext = 0.5*(mean(f(g)) + mean(f(~g))); done = abs(T - Tnext) < 0.5; T = Tnext; end MATLAB graythresh( ): Otsu [1979] * Histogram represented by a discrete probability density function:

1 ..., 2, 1, ,0 ,)( −== Lqnn

rp qqr

where n: total number of pixels, nq: number of pixels with graylevel rq, and L: number of intensity levels

* Suppose threshold k is chosen, two set of pixels: C0 (< k) and C1 (≥ k) * Define between-class variance,

2Bσ :

20110

211

200

2 )()()( µµωωµµωµµωσ −=−+−= TTB , where

∑−

=

=1

00 )(

k

qqq rpω , ∑

−

=

=1

1 )(L

kqqq rpω ,

10-12

∑−

=

=1

000 /)(

k

qqq rqp ωµ , ∑

−

=

=1

11 /)(L

kqqq rqp ωµ , and ∑

−

=

=1

0)(

L

qqqT rqpµ

* Objective: find k such that σ2B is maximized

If σ2B is maximized, within-class variance,

2Wσ , and total variance of

levels, 2Tσ , are also maximized:

211

200

2 σωσωσ +=W , ∑ =−=

L

i iTT pi1

22 )( µσ

((cc)) RReeggiioonn--bbaasseedd sseeggmmeennttaattiioonn - Region-based segmentation

Partition an image into regions - Basic formula

Let R represent the entire image Segmentation: partition R into n subregions, R1, R2, …, Rn, such that

10-13

* RRn

ii =

=U

1

Segmentation must be complete * Ri is a connected region, i = 1, 2, …n

Points in a region must be connected * Ri ∩ Rj = ∅, for all i and j, i ≠ j

Regions must be disjoint * P(Ri) = TRUE, i = 1, 2, …n

Pixels in a region must satisfy some properties (e.g., similar color) # P(Ri): a logical predicate defined on every point in Ri

* P(Ri ∪ Rj) = FALSE for adjacent regions Ri and Rj Ri and Rj are different in the sense of predicate P

- Region growing Group pixels or subregions into larger regions based on predefined criteria for growth

Method: 1. Start with a set of “seed” points

10-14

2. Grow regions by appending to each seed those neighboring pixels that have predefined properties similar to the seed (e.g., graylevel, texture…)

* E.g, points (3, 2) and (3, 4) are seeds; property: graylevel difference < 3: 0 0 3 6 7 a a a b b 1 1 2 8 7 a a a b b 0 1 6 7 7 a a b b b 2 0 7 6 6 a a b b b 0 1 5 6 5 a a b b b

Problems: how to select seeds and define properties - Region splitting and merging

Partion image into disjoint subregions and then perform splitting and merging

Method: subdivide R until subregion P(Ri) = TRUE 1. Splitting: if P(R) = FALSE, divide R into quadrants (Ri, i = 1, 2, …, 4);

if P(Ri) = FALSE, divide Ri into subquadrants, and so on Quadtree of quadregions

# MATLAB quadtree decomposition: qtdecomp( )

10-15

2. Merging: two adjacent regions Rj and Rk are merged if P(Rj ∪ Rk) = TRUE stop if no further merging is possible

Problem: translation of the object results in a different quadtree

R2

R4 R3

R14

R12R11

R13

R11 R12 R13 R14

R1 R2 R3 R4

R

R4 R3

R14

R13

R11 R12 R13 R14

R1 R2 R3 R4

R

1

1 1 0 0

0 0

Splitting:

Merging: R1 R2

10-16

((dd)) CCaannnnyy eeddggee ddeetteeccttoorr - Criteria for edge detection

Low error rate of detection: all edges should be found and nothing but edges

Localization of edges: distance between actual edges in the image and edges found should be minimized

Single response: multiple edge pixels should not be returned when only a single edge exists

R4 R3

R1

R1 R2 R3 R4

R

1 0 0

R2

0 R4 R3

R1

R2

R4

?

10-17

- Canny edge detection algorithm [1986] 1. Gaussian smoothing followed by derivative of Gaussian

Smooth image, x, and then find possible candidate pixels * To reduce computational cost, the filter can be applied to columns first

and then to rows (Gaussian filter is separable): # Create a 1D Gaussian filter g # Create a 1D filter dg according to the following equation

)2

exp( 2

2

2 σσxx

−− # Convolve g with dg to obtain gdg # Apply gdg to input image x producing x1 # Apply (gdg)' to x producing x2

# Edge image (magnitude): 22

21 xxxe +=

2. Nonmaximum suppression * Thresholding xe to remove pixels with low edge magnitude will produce

multiple edge responses

10-18

* Compute the edge direction of pixel p (i.e., edge gradient) Edge gradient: xg = tan–1(x2/x1)

* For p to be considered as a true edge pixel, p must have a greater magnitude than its neighbors in the edge direction (both ends) # Approach 1 (taking the following figure as an example)

- Compute interpolated magnitudes (m1) of a and b - Compute interpolated magnitudes (m2) of c and d - If xe(p) < m1 or xe(p) < m2, delete p

p

a

b

xg(p) = 30°

c

d

45°

0°

90°135°

22.5°

Approach 1 Approach 2

10-19

# Approach 2: - Quantize xg(p) to 0°, 45°, 90°, or 135° - If xe(p) < xe(a) or xe(p) < xe(d), delete p

3. Hysteresis thresholding * Thresholding with a single value is not appropriate because edges tend to

be broken * Two threshold values: a low value tL and a high value tH * If xe(p) < tL, p is deleted (nonedge pixel) * If xe(p) ≥ tH, p is marked as an edge pixel (edge pixel) * If tL ≤ xe(p) < tH (possible edge pixel) and if starting from p and

following other possible edge pixels can lead to an edge pixel, p is also marked as an edge pixel

10-20

((ee)) HHaarrrriiss eeddggee//ccoorrnneerr ddeetteeccttoorr - Moravec interest point detector

Measure directional variance over a small window (auto-correlation):

E(x, y) = Σu,v {Ix+u,y+v – Iu,v}2, = Σi=1~9{Pi – Qi}2

where I is the image, (x, y) is the shift of the window, and the center of the

Edge

Edge

Possible edge

Edge

Edge

Possible edge Nonedge

10-21

window is regarded as the origin

Define 8 shift directions: x, y = –1, 0, 1, excluding (0, 0), total response:

∑≠−=

=

)0,0(),(1,0,1,

),( yx

yxyxEE

Window of (u, v), u, v = –1, 0, 1

(0,0)

(x, y) u

v 1 2 3

654

7 8 9

P

Q

10-22

If the center is in a smooth local area, E will be small If E ≥ T, an interesting point is found

- Harris edge/corner detector Moravec’s measure is noisy due to the binary window (1’s within the window and 0’s elsewhere) * Improvement: use a smooth circular window, e.g, a Gaussian window

wu,v = exp –(u2+v2)/2σ2

E(x, y) = Σu,v wu,v{Ix+u,y+v – Iu,v}2

* Typically, a 3×3 Gaussian with σ = 0.5 is

=

=

0113.00838.00113.00838.06193.00838.00113.00838.00113.0

987

654

321

wwwwwwwww

w

Moravec’s measure is anisotropic (only 8 discrete directions) * Improvement: use the Taylor series to represent Ix+u,y+v to achieve

10-23

isotropy; Taylor expansion of a 2D function f about a point (x, y): { }

{ } L+∆+∆∆+∆

+∆+∆+=∆+∆+22 ),(),(2),(

!21

),(),(),(),(

yyxfyxyxfxyxf

yyxfxyxfyxfyyxxf

yyxyxx

yx

* Hence, E(∆x, ∆y) = Σu,v wu,v {Iu,v + ∆xX +∆yY + O(∆x2, ∆y2) + … – Iu,v}2

= Σu,v wu,v {∆xX +∆yY + O(∆x2, ∆y2) + …}2,

where the derivatives are approximated by X = I ⊗ [–1, 0, 1] = ∂I/∂x Y = I ⊗ [–1, 0, 1]T = ∂I/∂y (⊗ : convolution)

* Denoting ∆x and ∆y by x and y, respectively, and ignoring high order terms:

E(x, y) = Σu,v wu,v {∆xX +∆yY}2 =

10-24

Σu,v wu,v {X2x2 + 2XYxy + Y2y2} = Ax2 + 2Cxy + By2,

where A = X2 ⊗ w, B = Y2 ⊗ w, and C = (XY) ⊗ w.

* Rewritten: E(x, y) = [x y] M [x y]T, where

=

BCCA

M

* A linear algebra problem: eigan values and eigan vectors

Mx = λx

where M is a matrix, x is a vector, and λ is a scalar A vector transformed by a matrix is equal to the vector multiplied by a scalar

# E.g.,

=

−

→

=

−

=2

1

2

1

2

1 4211

,4211

xx

xx

xx

M λx

0)4(20)1(

or 42 21

21

221

121

=−+=−−

=+−=+

xxxx

xxxxxx

λλ

λλ

λ1 = 3, λ2 = 2: eigan values with corresponding eigan vectors

10-25

* Geometric meaning:

Define the corner response by matrix determinant and trace (avoid computing eigenvalues):

R = Det(M) – k⋅Tr(M)2 = (AB–C2) – k(A+B)

(λmax)-1/2

Direction of the fastest change

Direction of the slowest change

(λmin)-1/2

10-26

where k ≈ 0.04 (Note: R = λ1λ2 – k(λ1 + λ2)) R depends only on eigenvalues of M * R is large for a corner * R is negative with large magnitude for an edge * |R| is small for a flat region Iso-response contours of R on the λ1λ2-plane:

R = R1 R = R2 R = R3

α

β

10-27

λ1

λ2

“Corner” λ1 and λ2 are large, λ1 ~ λ2; E increases in all directions

λ1 and λ2 are small;E is almost constant in all directions

“Edge” λ1 >> λ2

“Edge” λ2 >> λ1

“Flat” region

Classification of image points using eigenvalues of M: