Advanced Digital Image P i d OthProcessing and Others
Xiaojun QiREU Sit P i CVMA-- REU Site Program in CVMA
(2010 Summer)
1
OutlineOutline• SegmentationSegmentation
– Strategies and Data StructuresO i f Al ith– Overview of Algorithms
• Region Splitting• Region Merging• Region Growing
• K-Means Algorithm Based Segmentation• K-Means Algorithm Based Segmentation
2
Importance of Segmentationpo ta ce o Seg e tat o• Segmentation is generally the first stage in
any attempt to analyze or interpret an image automatically.g y
• Segmentation bridges the gap between g g g plow-level image processing and high-level image processing.g p g
• Some kinds of segmentation technique will g qbe found in any application involving the detection, recognition, and measurement
3
, g ,of objects in images.
Importance of Segmentation p g(Cont.)
• The role of segmentation is crucial in most tasks requiring image analysis. Thetasks requiring image analysis. The success or failure of the task is often a direct consequence of the success ordirect consequence of the success or failure of segmentation.
• However, a reliable and accurate t ti f i i i lsegmentation of an image is, in general,
very difficult to achieve by purely 4
automatic means.
Image Segmentation ExampleImage Segmentation Example
5
Image Segmentation-- Descriptive Definition
• Segmentation subdivides an image into its constituent g gregions or objects. That is, it partitions an image into distinct regions that are meant to correlate strongly with bj t f t f i t t i th iobjects or features of interest in the image.
S t ti l b d d f• Segmentation can also be regarded as a process of grouping together pixels that have similar attributes.
• The level to which the subdivision is carried depends on the problem being solved That is segmentationon the problem being solved. That is, segmentation should stop when the objects of interest in an application have been isolated. There is no point in
6carrying segmentation past the level of detail required to identify those elements.
7
8
Image Segmentationg g-- A Math Oriented Descriptive Definition
• It is the process that partitions the image pixels into non-overlapping regionsp pp g gsuch that:
1. Each region is homogeneous (i.e., uniform in terms of the pixel attributes such as intensityterms of the pixel attributes such as intensity, color, range, or texture, and etc.) and connectedconnected.
2. The union of adjacent regions is not homogeneous
9
homogeneous.
Image Segmentationg g-- Explanation
1. All pixels must be assigned to regions.2 Each pixel must belong to a single region2. Each pixel must belong to a single region
only.3 E h i t b if3. Each region must be uniform.4. Any merged pair of adjacent regions y e ged pa o adjace t eg o s
must be non-uniform.5 E h i t b t d t f5. Each region must be a connected set of
pixels.10
11How many regions do these two pictures have?
How many regions do these two pictures have?
12
Image Segmentation StrategiesImage Segmentation Strategies• Image segmentation algorithms generallyImage segmentation algorithms generally
are based on one of two basic properties of intensity values: discontinuity and similarityintensity values: discontinuity and similarity.
• Discontinuity based approach: Partition an image based on abrupt changes in intensityimage based on abrupt changes in intensity.
• Similarity based approach: Partition an image based on regions that are similar according to a set of predefined criteria.– Thresholding– Region splitting and merging
13
g p g g g– Region growing
Discontinuity vs. SimilarityDiscontinuity vs. Similarity• Techniques based on discontinuity attempt to
partition the image by detecting abrupt changes inpartition the image by detecting abrupt changes in gray level. Point, line, and edge detectors.
• Techniques based on similarity attempt to create• Techniques based on similarity attempt to create the uniform regions by grouping together connected pixels that satisfy predefined similarityconnected pixels that satisfy predefined similarity criteria. Therefore, the results of segmentation may depend critically on these criteria and on thedepend critically on these criteria and on the definition of connectivity.Th h b d di ti it d• The approaches based on discontinuity and similarity mirror one another in the sense that
l ti f b d i i l t t b ki14
completion of a boundary is equivalent to breaking one region into two.
Similarity Based Approachy pp-- ThresholdingThis
histogram-based
How to choose threshold inapproach
assumes that different
threshold in this case?
different features in an image give rise to distinctrise to distinct peaks in its histogram.
Case A: The image is composed of one light object on a dark background, in such a way that object and background pixels have gray levels grouped into two dominant modes.
15
have gray levels grouped into two dominant modes.
Case B: Two types of light objects on a dark background.
Thresholding Ex 1es o d g
16
Ex 2
Importance of accurate threshold selection (a) Input Image (b)17
Importance of accurate threshold selection. (a) Input Image (b) Correct choice of threshold (T = 90) (c) Threshold too low (T = 40) (d) Threshold too high (T = 215)
Determination of Threshold T-- An Iterative Approach
1. Select an initial estimate for T.2. Segment the image using T. This will produce
two groups of pixels: G1 consisting of all pixels ith l l l > T d G2 i ti fwith gray level values > T and G2 consisting of
pixels with gray level values <=T.3 Compute the average gray levels µ1 and µ 23. Compute the average gray levels µ1 and µ 2
for the pixels in regions G1 and G2.4. Compute a new threshold value: T = (µ1+4. Compute a new threshold value: T (µ1
µ2)/2.5. Repeat step 2 through 4 until the difference in p p g
Ts in successive iterations is smaller than a predefined parameter T0.
18How do you select an initial estimate of T?
Similarity Based Approach-- Region Splitting
Basic Idea: Divide an image into smaller andsmaller regions until all the pixels in the differentg pregions satisfy the predefined uniformity predicateor uniformity measure for that regionor uniformity measure for that region.Steps:• It begins with the entire image in one region.• The region are then split to form sub-regions
which satisfy the basic segmentation criterion using a suitable uniformity predicate or
19uniformity measure.
Special Data Structuresp-- Picture Tree in Region Splitting
• A picture tree is a graph structure in the• A picture tree is a graph structure in the form of a tree that has an arc between two
(nodes (i.e., R1 and R2, where they respectively represent two regions) of the p y p g )graph, if R2 is contained in R1 (i.e., R1 is the parent node of R2)the parent node of R2).
• It indicates that R2 is a sub-region of the i R1 R1region R1. R1
20R4R3R2
Special Data StructuresQ ad Pict re Tree (QPT)-- Quad Picture Tree (QPT)
in Region Splittingg p g• Quad picture tree (QPT): It is a picture tree with
th i i l i th t t i dthe original image as the start region and progressively divides each region into four ( ) b i ith l b f(square) sub-regions with an equal number of pixels.
• The QPT can be used to guide the search for regions with uniform gray-levels for gray-level images by developing a measure of uniformity (homogeneity) to test the regions as candidates
21to be split.
QPT ExampleQPT ExampleRQ d Pi t T R1 R2 R3 R4
R11 R12 R13 R14
Quad Picture Tree
R11 R12 13 14
R111R112R113 RImage Data
R111R112R113 R114 01 2
3
0 0K P01 2 3 0 1 2 3
0 1 2 3 0 1 2 3E J
K
L M N O
P
1 2 3 0 1 2 3
A B C D F G H I
001(B) 002(C) 003(D) 01 (E) 020(F) 021(G)22
Partition into uniform regions
001(B), 002(C), 003(D), 01x(E), 020(F), 021(G), 023(I), 03x(J), 1XX(K), 21X(M), 23X(O)
One Common U if it (Si il it ) MUniformity (Similarity) Measure
Determine whether the region should be splitC id th f i R lit i t th
R1 R2• Consider the case of region R split into the subregions R1, R2, R3, and R4. Let µ be the mean of R and µi be the mean of Ri
1 2
R3 R4the mean of R and µi be the mean of Ri.
∑ −4
2)( i μμWhat is the formula to calculate
the variance?
∑
∑== 4
2
1)(i
ii
RUσ ∑
−
−==1
222 )()()()(
L
ii rpmrrr σμ
• If U(R) is large then R is not uniform and
∑=1i
iσ ∑=0
2 ).()()()(i
ii rpmrrr σμ
If U(R) is large, then R is not uniform and should be split into the four sub-regions.
• It indicates that the sub regions R differ23
• It indicates that the sub-regions Ri differ from R but are themselves uniform.
One Examplepµ = 3, µ1 = 2, µ2 = 3, µ3 = 2, µ4 = 5.
)2)(( 2−∑ pg2 2 4 42 2 2 2
;04
))((
2
121 ==
∑
∑∈
pgRpσ
1 1 5 53 3 5 5
;14
11114
)3)(( 2
222 =
+++=
−=∑∈
pgRpσ
3 3 5 5
;11111)2)((
442
32 =+++
=−
=∑∈
pgRpσ
)5)((
;144
2
3
−
===
∑ pg
σR1 R2
R3 R4
.04
424 ==∑∈Rpσ
6)35()32()33()32( 2222 +++243
26
0110)35()32()33()32()( ==
+++−+−+−+−
=RU
Similarity Based Approach-- Region Merging
1. Define a splitting method to segment the image into small atomic regions satisfying the basic
t ti it i i it blsegmentation criterion using a suitable uniformity predicate.
2 D fi th d f i dj t i2. Define a method for merging adjacent regions. That is, merge two adjacent regions which satisfy the merging conditions and the basicsatisfy the merging conditions and the basic segmentation criterion (i.e., the uniform predicate for the union of these two adjacentpredicate for the union of these two adjacent regions is true).
3 Repeat the merging procedure If no regions25
3. Repeat the merging procedure. If no regions can be merged, then stop.
Special Data StructuresRegion Adjacency Graph (RAG)-- Region Adjacency Graph (RAG)
in Region Mergingg g g• Region adjacency graph gives the region adjacency
relations. It is a structure often used with the QPT. That is, region R1 is adjacent to region R2 if there is a pixel in R1 with a 4-neighbor, or 8-neighbor, or m-neighbor in Rneighbor in R2.
0 0 0 0 0 02
0 1 1 4 4 00 1 1 3 0 0
E
1
0 1 1 3 0 00 0 2 0 0 0
10
26
0 0 2 0 0 00 0 2 0 0 0
34
Region Adjacency Graph
Similarity-Based ApproachR i G i-- Region Growing
• Region growing is a bottom-up procedureRegion growing is a bottom up procedure that starts with a set of seed pixels. The aim is to grow a uniform connected regionaim is to grow a uniform, connected region from each seed. A pixel is added to a region if and only ifregion if and only if– It has not been assigned to any other region
It is a neighbor of that region– It is a neighbor of that region– The new region created by addition of the
pixel is still uniformpixel is still uniform.• In general, it starts with a single pixel
( d) d dd i l l l27
(seed) and add new pixels slowly.
Seeded Region Growingg g-- Non-Math Perspective
Basic Ideas:1 Choose the seed pixels1. Choose the seed pixels.2. Check the neighboring pixels and add
th t th i if th i il tthem to the region if they are similar to the seed by using a certain predicate.
3. Repeat step 2 for each of the newly added pixels; stop if no more pixels canadded pixels; stop if no more pixels can be added.
28
Seeded Region Growing Notations
• Let T be the set of pixels that are neighbors to some p gatomic region. That is:
U I U ii ApNandAppT }))()((|{ φ≠∉=
where N(p) is the 8- or 4 or m-neighbor region of pixel p. The set of Ai are initially the atomic regions.
i i
• If pixel p is in T, then for every atomic region Ai such that
|)}({mean)(|)(letthen,)( iii AgpgppNA −=Φ≠∩ δ
This is the difference between p and Ai, which is computed as the difference between the gray-level of p and the average gray level in A
29
of p and the average gray-level in Ai.
Seeded Region Growing NotationsSeeded Region Growing Notations
• The minimum of these differences over Ai is defined as })(|)(min{)( φδδ ≠∩= ii ApNpp
• The point we are interested in is:The point we are interested in is:
}minimumais)(|{' pTpp δ∈=
30
Seeded Region Growing AlgorithmMath Perspective-- Math Perspective
1. Label the pixels in the atomic regions with their initial labeling.2. Find the pixels in T. Assign each pixel in T to a temporary label2. Find the pixels in T. Assign each pixel in T to a temporary label
based on its neighboring region. Put these pixels in a linear list, which is sorted in an ascending order according to .
3 R th fi t i l f th li li t hil th li t i t)( piδ
3. Remove the first pixel p from the linear list while the list is notempty.
3.1 Set p to the same label as its neighboring region. 3.2 Update the mean of the affected (i.e., expanded) region.3 3 If the new neighbors of the expanded region already have3.3 If the new neighbors of the expanded region already have temporary labels, label these neighbors as boundary pixels. These boundary pixels will not be carried on to the further processprocess.3.4 Otherwise, assign the new neighbors a temporary label based on their neighboring region.3 5 R l l t th f h t il l b l d)(δ
31
3.5 Recalculate the for each temporarily labeled neighboring pixel of the expanded region. Order the linear list in ascending order according to the new .
)( piδ
)( piδ
1 1 3 4 1 1 3 4 1 1 3 4 1 1 3 4Ave1 = 1, Ave2 = 7 Ave1 = 1, Ave2 = 7 Ave1 = 1, Ave2 = 7 Ave1 = 1, Ave2 = 7
1 1 3 41 3 5 62 3 6 7
1 1 3 41 3 5 62 3 6 7
1 1 3 41 3 5 62 3 6 7
1 1 3 41 3 5 62 3 6 72 3 6 7
2 3 6 72 3 6 72 3 6 7
2 3 6 72 3 6 7
2 3 6 72 3 6 7
Ave1 1 25 Ave2 7 Ave1 1 4 Ave2 70 0 0 1 1 1 2 2 0 0 1 1 1 1 2 2 2 0 1 1 1 1 2 2 2 1 1 1 1 2 2 2
Ave1 1 4 Ave2 6 7Ave1 = 1.4, Ave2 = 6.5
1 1 3 4 1 1 3 4
Ave1 = 1.25, Ave2 = 7 Ave1 = 1.4, Ave2 = 7
1 1 3 4 1 1 3 4
Ave1 = 1.4, Ave2 = 6.7
1 3 5 62 3 6 7
1 3 5 62 3 6 7
1 3 5 62 3 6 7
1 3 5 62 3 6 7 2 3 6 7
2 3 6 72 3 6 72 3 6 7
2 3 6 72 3 6 7
2 3 6 72 3 6 7
320.75 1 1 1 1.75 1.75 1.75 1.75
1 1 1 1.6 1.6 1.6 1.6 0.7 0.7 1.6 0.5 2.5
Region-Merging-BasedImage Segmentation Illustration
33
Region-Merging-Based Segmentation Using Color Features
• Divide the image into 2x2 non-overlapping blocks.
• Extract a color feature vector (i.e., the mean color of the block) for each block by using the ) y gLuv color space.
• Apply an unsupervised K-means algorithm toApply an unsupervised K means algorithm to cluster the color features.
• The segmentation process adaptively increases• The segmentation process adaptively increases the number of regions C (initially set as 2) until a termination criterion is satisfied
34
termination criterion is satisfied.
Cluster Seeking-- K-Means Algorithm
• The K-means algorithm is based on the minimization of a performance index which pis defined as the sum of the squared distances from all points in a clusterdistances from all points in a cluster domain to the cluster center.
35
Cluster Seeking-- K-Means Algorithm (Cont.)
• Step 1: Choose K initial cluster centers z1(1), z2(1), …, zK(1). These are arbitrary and are usually selected as the first K samples of the given sample setthe first K samples of the given sample set.
• Step 2: At the kth iterative step distribute the samples• Step 2: At the kth iterative step, distribute the samples {x} among the K cluster domains, using the relation,
For all i = 1 2 K i ≠ j where Sj(k) denotes the set
)()( if )( kkkS ijj zxzxx −<−∈ (1)
For all i = 1, 2, . . . , K, i ≠ j, where Sj(k) denotes the set of samples whose cluster center is zj(k). Ties in expression (1) are resolved arbitrarily.
36
p ( ) y
• Step 3: From the results of Step 2, compute the new cluster centers zj(k+1), j = 1, 2, . . . , K, such that the j( ), j , , , ,sum of the squared distances from all points in Sj(k) to the new cluster center is minimized. In other words, th l t t j(k 1) i t d th t ththe new cluster center zj(k+1) is computed so that the performance index
2K , 2, ,1 ,)1(
2
)(
K=+−= ∑∈
jkJkS
jjjx
zx
is minimized. The zj(k+1), which minimizes this performance index is simply the sample means of Sj(k) Th f th l t t i i bSj(k). Therefore, the new cluster center is given by
K211)1( ==+ ∑ jk xz
Where Nj is the number of samples in Sj(k) . The
K,2, ,1 ,)1()(
K==+ ∑∈
jN
kkSj
jjx
xz
37
j p j( )name “K-means” is obviously derived from the manner in which cluster centers are sequentially updated.
• Step 4: If zj(k+1) = zj(k) for j = 1, 2, . . . , K, the l ith h d d th d ialgorithm has converged and the procedure is
terminated. Otherwise go to Step 2.
• The behavior of the K-means algorithm is ginfluenced by:– The number of cluster centers specified;The number of cluster centers specified;– The choice of initial cluster centers; – The order in which the samples are taken;The order in which the samples are taken; – The geometrical properties of the data.
• In most practical cases the application of this• In most practical cases the application of this algorithm will require experimenting with various values of K as well as different choices of starting
38
values of K as well as different choices of starting configurations.
Cluster SeekingK Means Algorithm Example-- K-Means Algorithm Example
39
• Step 1 Let K = 2 and choose z1(1) = x1 = (0• Step 1. Let K = 2 and choose z1(1) = x1 = (0, 0)’, z2(1) = x2 = (1, 0)’.
• Step 2. Since and)1()1( 111 izxzx −<−
h th t Si il l th
2,)1()1( 313 =−<− iizxzx
}{)1(Swe have that . Similarly, the remaining patterns are closer to z2(1), so
},{)1( 311 xx=S
},,,,{)1( 205422 xxxx K=S
40
• Step 3. Update the cluster centers:
= ∑1)2(1 xzN
+=
∈
)(21
31
)1(1 1
xx
x SN
⎟⎟⎠
⎞⎜⎜⎝
⎛=
500.0
2 31
=
⎟⎠
⎜⎝
∑1)2(
5.0
2 xz
+++=
∑∈
)(1
)2()1(2
22
xxx
xzx SN
⎟⎟⎞
⎜⎜⎛
+++=
67.5
)(18 2042 xxx L
41⎟⎟⎠
⎜⎜⎝
=33.5
• Step 4. Since zj(2) ≠ zj(1), for j = 1, 2, we return p j( ) j( ), j , ,to Step 2.
• Step 2. With the new cluster centers we obtain)2()2( < zxzx
8,,2,1for )2()2( 21
K=
−<−
lll zxzx
)2()2( and
12 −<− ll zxzx
• Therefore: 20,,10,9for K=l
}{)2( xxxSand
},,,{)2( 8211 xxx K=S
42},,,{)2( 201092 xxx K=S
• Step 3. Update the cluster centers:1
= ∑∈
1)3()2(1
11
xzx SN
++= )(81
821 xxx L
⎟⎟⎠
⎞⎜⎜⎝
⎛=
13125.1
=
⎠⎝
∑1)3(
13.1
2 xz
+++
∑∈
)(1
)()2(2
22
xxx
x SN
⎟⎞
⎜⎛
+++=
67.7
)(12 20109 xxx L
43⎟⎟⎠
⎞⎜⎜⎝
⎛=
33.767.7
• Step 4. Since zj(3) ≠ zj(2), for j = 1, 2, we return to Step 2.
• Step 2 yields the same results as in the previous iteration:)3()4(d)3()4( SSSS
• Step 3 also yields the same results.
)3()4(and)3()4( 2211 SSSS ==
Step 4 Since zj(4) = zj(3) for j = 1 2 the algorithm has• Step 4. Since zj(4) = zj(3) for j = 1, 2, the algorithm has converged, yielding these cluster centers:
⎞⎛⎞⎛ 677251⎟⎟⎠
⎞⎜⎜⎝
⎛=⎟⎟
⎠
⎞⎜⎜⎝
⎛=
33.767.7
,13.125.1
21 zz
These results agree with what we would expect from inspection of the given data.
44