Advanced Digital Image P i d OthProcessing and Others

Xiaojun QiREU Sit P i CVMA-- REU Site Program in CVMA

(2010 Summer)

1

OutlineOutline• SegmentationSegmentation

– Strategies and Data StructuresO i f Al ith– Overview of Algorithms

• Region Splitting• Region Merging• Region Growing

• K-Means Algorithm Based Segmentation• K-Means Algorithm Based Segmentation

2

Importance of Segmentationpo ta ce o Seg e tat o• Segmentation is generally the first stage in

any attempt to analyze or interpret an image automatically.g y

• Segmentation bridges the gap between g g g plow-level image processing and high-level image processing.g p g

• Some kinds of segmentation technique will g qbe found in any application involving the detection, recognition, and measurement

3

, g ,of objects in images.

Importance of Segmentation p g(Cont.)

• The role of segmentation is crucial in most tasks requiring image analysis. Thetasks requiring image analysis. The success or failure of the task is often a direct consequence of the success ordirect consequence of the success or failure of segmentation.

• However, a reliable and accurate t ti f i i i lsegmentation of an image is, in general,

very difficult to achieve by purely 4

automatic means.

Image Segmentation ExampleImage Segmentation Example

5

Image Segmentation-- Descriptive Definition

• Segmentation subdivides an image into its constituent g gregions or objects. That is, it partitions an image into distinct regions that are meant to correlate strongly with bj t f t f i t t i th iobjects or features of interest in the image.

S t ti l b d d f• Segmentation can also be regarded as a process of grouping together pixels that have similar attributes.

• The level to which the subdivision is carried depends on the problem being solved That is segmentationon the problem being solved. That is, segmentation should stop when the objects of interest in an application have been isolated. There is no point in

6carrying segmentation past the level of detail required to identify those elements.

7

8

Image Segmentationg g-- A Math Oriented Descriptive Definition

• It is the process that partitions the image pixels into non-overlapping regionsp pp g gsuch that:

1. Each region is homogeneous (i.e., uniform in terms of the pixel attributes such as intensityterms of the pixel attributes such as intensity, color, range, or texture, and etc.) and connectedconnected.

2. The union of adjacent regions is not homogeneous

9

homogeneous.

Image Segmentationg g-- Explanation

1. All pixels must be assigned to regions.2 Each pixel must belong to a single region2. Each pixel must belong to a single region

only.3 E h i t b if3. Each region must be uniform.4. Any merged pair of adjacent regions y e ged pa o adjace t eg o s

must be non-uniform.5 E h i t b t d t f5. Each region must be a connected set of

pixels.10

11How many regions do these two pictures have?

How many regions do these two pictures have?

12

Image Segmentation StrategiesImage Segmentation Strategies• Image segmentation algorithms generallyImage segmentation algorithms generally

are based on one of two basic properties of intensity values: discontinuity and similarityintensity values: discontinuity and similarity.

• Discontinuity based approach: Partition an image based on abrupt changes in intensityimage based on abrupt changes in intensity.

• Similarity based approach: Partition an image based on regions that are similar according to a set of predefined criteria.– Thresholding– Region splitting and merging

13

g p g g g– Region growing

Discontinuity vs. SimilarityDiscontinuity vs. Similarity• Techniques based on discontinuity attempt to

partition the image by detecting abrupt changes inpartition the image by detecting abrupt changes in gray level. Point, line, and edge detectors.

• Techniques based on similarity attempt to create• Techniques based on similarity attempt to create the uniform regions by grouping together connected pixels that satisfy predefined similarityconnected pixels that satisfy predefined similarity criteria. Therefore, the results of segmentation may depend critically on these criteria and on thedepend critically on these criteria and on the definition of connectivity.Th h b d di ti it d• The approaches based on discontinuity and similarity mirror one another in the sense that

l ti f b d i i l t t b ki14

completion of a boundary is equivalent to breaking one region into two.

Similarity Based Approachy pp-- ThresholdingThis

histogram-based

How to choose threshold inapproach

assumes that different

threshold in this case?

different features in an image give rise to distinctrise to distinct peaks in its histogram.

Case A: The image is composed of one light object on a dark background, in such a way that object and background pixels have gray levels grouped into two dominant modes.

15

have gray levels grouped into two dominant modes.

Case B: Two types of light objects on a dark background.

Thresholding Ex 1es o d g

16

Ex 2

Importance of accurate threshold selection (a) Input Image (b)17

Importance of accurate threshold selection. (a) Input Image (b) Correct choice of threshold (T = 90) (c) Threshold too low (T = 40) (d) Threshold too high (T = 215)

Determination of Threshold T-- An Iterative Approach

1. Select an initial estimate for T.2. Segment the image using T. This will produce

two groups of pixels: G1 consisting of all pixels ith l l l > T d G2 i ti fwith gray level values > T and G2 consisting of

pixels with gray level values <=T.3 Compute the average gray levels µ1 and µ 23. Compute the average gray levels µ1 and µ 2

for the pixels in regions G1 and G2.4. Compute a new threshold value: T = (µ1+4. Compute a new threshold value: T (µ1

µ2)/2.5. Repeat step 2 through 4 until the difference in p p g

Ts in successive iterations is smaller than a predefined parameter T0.

18How do you select an initial estimate of T?

Similarity Based Approach-- Region Splitting

Basic Idea: Divide an image into smaller andsmaller regions until all the pixels in the differentg pregions satisfy the predefined uniformity predicateor uniformity measure for that regionor uniformity measure for that region.Steps:• It begins with the entire image in one region.• The region are then split to form sub-regions

which satisfy the basic segmentation criterion using a suitable uniformity predicate or

19uniformity measure.

Special Data Structuresp-- Picture Tree in Region Splitting

• A picture tree is a graph structure in the• A picture tree is a graph structure in the form of a tree that has an arc between two

(nodes (i.e., R1 and R2, where they respectively represent two regions) of the p y p g )graph, if R2 is contained in R1 (i.e., R1 is the parent node of R2)the parent node of R2).

• It indicates that R2 is a sub-region of the i R1 R1region R1. R1

20R4R3R2

Special Data StructuresQ ad Pict re Tree (QPT)-- Quad Picture Tree (QPT)

in Region Splittingg p g• Quad picture tree (QPT): It is a picture tree with

th i i l i th t t i dthe original image as the start region and progressively divides each region into four ( ) b i ith l b f(square) sub-regions with an equal number of pixels.

• The QPT can be used to guide the search for regions with uniform gray-levels for gray-level images by developing a measure of uniformity (homogeneity) to test the regions as candidates

21to be split.

QPT ExampleQPT ExampleRQ d Pi t T R1 R2 R3 R4

R11 R12 R13 R14

Quad Picture Tree

R11 R12 13 14

R111R112R113 RImage Data

R111R112R113 R114 01 2

3

0 0K P01 2 3 0 1 2 3

0 1 2 3 0 1 2 3E J

K

L M N O

P

1 2 3 0 1 2 3

A B C D F G H I

001(B) 002(C) 003(D) 01 (E) 020(F) 021(G)22

Partition into uniform regions

001(B), 002(C), 003(D), 01x(E), 020(F), 021(G), 023(I), 03x(J), 1XX(K), 21X(M), 23X(O)

One Common U if it (Si il it ) MUniformity (Similarity) Measure

Determine whether the region should be splitC id th f i R lit i t th

R1 R2• Consider the case of region R split into the subregions R1, R2, R3, and R4. Let µ be the mean of R and µi be the mean of Ri

1 2

R3 R4the mean of R and µi be the mean of Ri.

∑ −4

2)( i μμWhat is the formula to calculate

the variance?

∑

∑== 4

2

1)(i

ii

RUσ ∑

−

−==1

222 )()()()(

L

ii rpmrrr σμ

• If U(R) is large then R is not uniform and

∑=1i

iσ ∑=0

2 ).()()()(i

ii rpmrrr σμ

If U(R) is large, then R is not uniform and should be split into the four sub-regions.

• It indicates that the sub regions R differ23

• It indicates that the sub-regions Ri differ from R but are themselves uniform.

One Examplepµ = 3, µ1 = 2, µ2 = 3, µ3 = 2, µ4 = 5.

)2)(( 2−∑ pg2 2 4 42 2 2 2

;04

))((

2

121 ==

∑

∑∈

pgRpσ

1 1 5 53 3 5 5

;14

11114

)3)(( 2

222 =

+++=

−=∑∈

pgRpσ

3 3 5 5

;11111)2)((

442

32 =+++

=−

=∑∈

pgRpσ

)5)((

;144

2

3

−

===

∑ pg

σR1 R2

R3 R4

.04

424 ==∑∈Rpσ

6)35()32()33()32( 2222 +++243

26

0110)35()32()33()32()( ==

+++−+−+−+−

=RU

Similarity Based Approach-- Region Merging

1. Define a splitting method to segment the image into small atomic regions satisfying the basic

t ti it i i it blsegmentation criterion using a suitable uniformity predicate.

2 D fi th d f i dj t i2. Define a method for merging adjacent regions. That is, merge two adjacent regions which satisfy the merging conditions and the basicsatisfy the merging conditions and the basic segmentation criterion (i.e., the uniform predicate for the union of these two adjacentpredicate for the union of these two adjacent regions is true).

3 Repeat the merging procedure If no regions25

3. Repeat the merging procedure. If no regions can be merged, then stop.

Special Data StructuresRegion Adjacency Graph (RAG)-- Region Adjacency Graph (RAG)

in Region Mergingg g g• Region adjacency graph gives the region adjacency

relations. It is a structure often used with the QPT. That is, region R1 is adjacent to region R2 if there is a pixel in R1 with a 4-neighbor, or 8-neighbor, or m-neighbor in Rneighbor in R2.

0 0 0 0 0 02

0 1 1 4 4 00 1 1 3 0 0

E

1

0 1 1 3 0 00 0 2 0 0 0

10

26

0 0 2 0 0 00 0 2 0 0 0

34

Region Adjacency Graph

Similarity-Based ApproachR i G i-- Region Growing

• Region growing is a bottom-up procedureRegion growing is a bottom up procedure that starts with a set of seed pixels. The aim is to grow a uniform connected regionaim is to grow a uniform, connected region from each seed. A pixel is added to a region if and only ifregion if and only if– It has not been assigned to any other region

It is a neighbor of that region– It is a neighbor of that region– The new region created by addition of the

pixel is still uniformpixel is still uniform.• In general, it starts with a single pixel

( d) d dd i l l l27

(seed) and add new pixels slowly.

Seeded Region Growingg g-- Non-Math Perspective

Basic Ideas:1 Choose the seed pixels1. Choose the seed pixels.2. Check the neighboring pixels and add

th t th i if th i il tthem to the region if they are similar to the seed by using a certain predicate.

3. Repeat step 2 for each of the newly added pixels; stop if no more pixels canadded pixels; stop if no more pixels can be added.

28

Seeded Region Growing Notations

• Let T be the set of pixels that are neighbors to some p gatomic region. That is:

U I U ii ApNandAppT }))()((|{ φ≠∉=

where N(p) is the 8- or 4 or m-neighbor region of pixel p. The set of Ai are initially the atomic regions.

i i

• If pixel p is in T, then for every atomic region Ai such that

|)}({mean)(|)(letthen,)( iii AgpgppNA −=Φ≠∩ δ

This is the difference between p and Ai, which is computed as the difference between the gray-level of p and the average gray level in A

29

of p and the average gray-level in Ai.

Seeded Region Growing NotationsSeeded Region Growing Notations

• The minimum of these differences over Ai is defined as })(|)(min{)( φδδ ≠∩= ii ApNpp

• The point we are interested in is:The point we are interested in is:

}minimumais)(|{' pTpp δ∈=

30

Seeded Region Growing AlgorithmMath Perspective-- Math Perspective

1. Label the pixels in the atomic regions with their initial labeling.2. Find the pixels in T. Assign each pixel in T to a temporary label2. Find the pixels in T. Assign each pixel in T to a temporary label

based on its neighboring region. Put these pixels in a linear list, which is sorted in an ascending order according to .

3 R th fi t i l f th li li t hil th li t i t)( piδ

3. Remove the first pixel p from the linear list while the list is notempty.

3.1 Set p to the same label as its neighboring region. 3.2 Update the mean of the affected (i.e., expanded) region.3 3 If the new neighbors of the expanded region already have3.3 If the new neighbors of the expanded region already have temporary labels, label these neighbors as boundary pixels. These boundary pixels will not be carried on to the further processprocess.3.4 Otherwise, assign the new neighbors a temporary label based on their neighboring region.3 5 R l l t th f h t il l b l d)(δ

31

3.5 Recalculate the for each temporarily labeled neighboring pixel of the expanded region. Order the linear list in ascending order according to the new .

)( piδ

)( piδ

1 1 3 4 1 1 3 4 1 1 3 4 1 1 3 4Ave1 = 1, Ave2 = 7 Ave1 = 1, Ave2 = 7 Ave1 = 1, Ave2 = 7 Ave1 = 1, Ave2 = 7

1 1 3 41 3 5 62 3 6 7

1 1 3 41 3 5 62 3 6 7

1 1 3 41 3 5 62 3 6 7

1 1 3 41 3 5 62 3 6 72 3 6 7

2 3 6 72 3 6 72 3 6 7

2 3 6 72 3 6 7

2 3 6 72 3 6 7

Ave1 1 25 Ave2 7 Ave1 1 4 Ave2 70 0 0 1 1 1 2 2 0 0 1 1 1 1 2 2 2 0 1 1 1 1 2 2 2 1 1 1 1 2 2 2

Ave1 1 4 Ave2 6 7Ave1 = 1.4, Ave2 = 6.5

1 1 3 4 1 1 3 4

Ave1 = 1.25, Ave2 = 7 Ave1 = 1.4, Ave2 = 7

1 1 3 4 1 1 3 4

Ave1 = 1.4, Ave2 = 6.7

1 3 5 62 3 6 7

1 3 5 62 3 6 7

1 3 5 62 3 6 7

1 3 5 62 3 6 7 2 3 6 7

2 3 6 72 3 6 72 3 6 7

2 3 6 72 3 6 7

2 3 6 72 3 6 7

320.75 1 1 1 1.75 1.75 1.75 1.75

1 1 1 1.6 1.6 1.6 1.6 0.7 0.7 1.6 0.5 2.5

Region-Merging-BasedImage Segmentation Illustration

33

Region-Merging-Based Segmentation Using Color Features

• Divide the image into 2x2 non-overlapping blocks.

• Extract a color feature vector (i.e., the mean color of the block) for each block by using the ) y gLuv color space.

• Apply an unsupervised K-means algorithm toApply an unsupervised K means algorithm to cluster the color features.

• The segmentation process adaptively increases• The segmentation process adaptively increases the number of regions C (initially set as 2) until a termination criterion is satisfied

34

termination criterion is satisfied.

Cluster Seeking-- K-Means Algorithm

• The K-means algorithm is based on the minimization of a performance index which pis defined as the sum of the squared distances from all points in a clusterdistances from all points in a cluster domain to the cluster center.

35

Cluster Seeking-- K-Means Algorithm (Cont.)

• Step 1: Choose K initial cluster centers z1(1), z2(1), …, zK(1). These are arbitrary and are usually selected as the first K samples of the given sample setthe first K samples of the given sample set.

• Step 2: At the kth iterative step distribute the samples• Step 2: At the kth iterative step, distribute the samples {x} among the K cluster domains, using the relation,

For all i = 1 2 K i ≠ j where Sj(k) denotes the set

)()( if )( kkkS ijj zxzxx −<−∈ (1)

For all i = 1, 2, . . . , K, i ≠ j, where Sj(k) denotes the set of samples whose cluster center is zj(k). Ties in expression (1) are resolved arbitrarily.

36

p ( ) y

• Step 3: From the results of Step 2, compute the new cluster centers zj(k+1), j = 1, 2, . . . , K, such that the j( ), j , , , ,sum of the squared distances from all points in Sj(k) to the new cluster center is minimized. In other words, th l t t j(k 1) i t d th t ththe new cluster center zj(k+1) is computed so that the performance index

2K , 2, ,1 ,)1(

2

)(

K=+−= ∑∈

jkJkS

jjjx

zx

is minimized. The zj(k+1), which minimizes this performance index is simply the sample means of Sj(k) Th f th l t t i i bSj(k). Therefore, the new cluster center is given by

K211)1( ==+ ∑ jk xz

Where Nj is the number of samples in Sj(k) . The

K,2, ,1 ,)1()(

K==+ ∑∈

jN

kkSj

jjx

xz

37

j p j( )name “K-means” is obviously derived from the manner in which cluster centers are sequentially updated.

• Step 4: If zj(k+1) = zj(k) for j = 1, 2, . . . , K, the l ith h d d th d ialgorithm has converged and the procedure is

terminated. Otherwise go to Step 2.

• The behavior of the K-means algorithm is ginfluenced by:– The number of cluster centers specified;The number of cluster centers specified;– The choice of initial cluster centers; – The order in which the samples are taken;The order in which the samples are taken; – The geometrical properties of the data.

• In most practical cases the application of this• In most practical cases the application of this algorithm will require experimenting with various values of K as well as different choices of starting

38

values of K as well as different choices of starting configurations.

Cluster SeekingK Means Algorithm Example-- K-Means Algorithm Example

39

• Step 1 Let K = 2 and choose z1(1) = x1 = (0• Step 1. Let K = 2 and choose z1(1) = x1 = (0, 0)’, z2(1) = x2 = (1, 0)’.

• Step 2. Since and)1()1( 111 izxzx −<−

h th t Si il l th

2,)1()1( 313 =−<− iizxzx

}{)1(Swe have that . Similarly, the remaining patterns are closer to z2(1), so

},{)1( 311 xx=S

},,,,{)1( 205422 xxxx K=S

40

• Step 3. Update the cluster centers:

= ∑1)2(1 xzN

+=

∈

)(21

31

)1(1 1

xx

x SN

⎟⎟⎠

⎞⎜⎜⎝

⎛=

500.0

2 31

=

⎟⎠

⎜⎝

∑1)2(

5.0

2 xz

+++=

∑∈

)(1

)2()1(2

22

xxx

xzx SN

⎟⎟⎞

⎜⎜⎛

+++=

67.5

)(18 2042 xxx L

41⎟⎟⎠

⎜⎜⎝

=33.5

• Step 4. Since zj(2) ≠ zj(1), for j = 1, 2, we return p j( ) j( ), j , ,to Step 2.

• Step 2. With the new cluster centers we obtain)2()2( < zxzx

8,,2,1for )2()2( 21

K=

−<−

lll zxzx

)2()2( and

12 −<− ll zxzx

• Therefore: 20,,10,9for K=l

}{)2( xxxSand

},,,{)2( 8211 xxx K=S

42},,,{)2( 201092 xxx K=S

• Step 3. Update the cluster centers:1

= ∑∈

1)3()2(1

11

xzx SN

++= )(81

821 xxx L

⎟⎟⎠

⎞⎜⎜⎝

⎛=

13125.1

=

⎠⎝

∑1)3(

13.1

2 xz

+++

∑∈

)(1

)()2(2

22

xxx

x SN

⎟⎞

⎜⎛

+++=

67.7

)(12 20109 xxx L

43⎟⎟⎠

⎞⎜⎜⎝

⎛=

33.767.7

• Step 4. Since zj(3) ≠ zj(2), for j = 1, 2, we return to Step 2.

• Step 2 yields the same results as in the previous iteration:)3()4(d)3()4( SSSS

• Step 3 also yields the same results.

)3()4(and)3()4( 2211 SSSS ==

Step 4 Since zj(4) = zj(3) for j = 1 2 the algorithm has• Step 4. Since zj(4) = zj(3) for j = 1, 2, the algorithm has converged, yielding these cluster centers:

⎞⎛⎞⎛ 677251⎟⎟⎠

⎞⎜⎜⎝

⎛=⎟⎟

⎠

⎞⎜⎜⎝

⎛=

33.767.7

,13.125.1

21 zz

These results agree with what we would expect from inspection of the given data.

44