The iterative convolution-thresholding method (ICTM) for imagesegmentation
Dong Wang (University of Utah)Xiao-Ping Wang (HKUST)
1/ 21
Introduction
I Image: f : Ω→ [0, 1]d
I Ω : domain of image (discrete pixels)I d : number of channels (e.g., gray image: d = 1, color image: d = 3.)
I Image segmentation: the process of partitioning a digital image into multiple segmentsm
find the optimal partition in the sense of minimizing a objective functional
2/ 21
Objective functionals
Intuitively, the optimal partition should obey:I In each segment, the feature should be “almost constant” or has no “big” jumps. The
partition should be consistent with the image itself.I The boundary of the partition should be “relatively smooth” (try to ignore some noise from
in the image).
Mathematically, in the objective function, we should haveI A quantity to measure the consistency of the image in each segment (Fidelity term)I A quantity to measure the smoothness of the boundary of each segment (Regularization
term)
Generally, for the n-segment (Ω1,Ω2, · · · ,Ωn) case, we write the general objective functionalinto
E =n∑
i=1
∫Ωi
Fi(f ,Θ1, . . . ,Θn) dx +n∑
i=1
λi|∂Ωi|
I Fi: choice of Fi depends on the features of the image or the types of imageI Θi: possible parameters or functions needed to describe Fi
I λi: constant parametersI |∂Ωi|: the perimeter of the segment Ωi
3/ 21
Example
In each segment, require the feature of the image is close to a constant
⇓Fi(f ,Ci) = |Ci − f |2 (Ci: average of f in the segment Ωi)
⇓Objective functional
E(Ω1, . . . ,Ωn,C1, . . . ,Cn) =∑n
i=1∫Ωi|Ci − f |2 dx + λi
∑ni=1 |∂Ωi| (Chan–Vese Model)
[Chan and Vese, IEEE Trans. Image Process., 2001, Vese and Chan, Int. J. Comput. Vis., 2002]
4/ 21
Representing the partition and expressing the objective functionalExisting numerical method:I Prime dual, split Bregmann, framelets, K-means clustering, · · ·I Phase-field based methodI Level set method
Level set methodThe discrete image is interpreted as a function defined on a continuous domain.
⇓The boundary of the segment is represented as the levelset of an auxiliary function ϕ.
⇓Every term in the objective function can be expressed using ϕ and the Heaviside function of ϕ.
⇓Variation w.r.t ϕ gives a partial differential equation (PDE)
⇓Solving the PDE to evolve the initial guess to the final result to get the segmentation
Benefits: Very easy to adapt to arbitrary modelsDisadvantages:I Solving complicated PDEsI Reinitialization (or another technique) is needed for the auxiliary function ϕI Small artificial time step for the stability reason (converges slow)I Sensitive to numbers of segments
5/ 21
Using characteristic functions (2-segment case)I The first segment Ω1 is denoted by its characteristic function u(x), i.e.,
u(x) :=
1 if x ∈ Ω1,
0 otherwise.
I the characteristic function of the second segment Ωc1 is 1− u(x).
I the interface between two segments is now implicitly represented by u(x).
I The first term in the objective function∫Ω1
F1 dx +∫Ω2
F2 dx =∫Ω uF1 + (1− u)F2 dx
I The perimeter term [Miranda et al., Annales de la faculte des sciences de ToulouseMathematiques, 2007]
|∂Ω1| ≈√π
τ
∫Ω
uGτ ∗ (1− u) dx
6/ 21
Approximation of the objective functional
E ≈ Eτ (u,Θ): = Ef (u,Θ) + Eτr (u,Θ)
I Ef (u,Θ) =∫Ω uF1(f ,Θ) + (1− u)F2(f ,Θ) dx
I Eτr (u,Θ) = λ√πτ
∫Ω uGτ ∗ (1− u) dx
Target: Find uτ,? such that
(uτ,?,Θτ,?) = arg minu∈B,Θ∈S
Eτ (u,Θ)
where B : = u ∈ BV(Ω,R) | u = 0, 1, S is the admissible set of Θ, and BV(Ω,R)denotes the bounded-variation functional space.
7/ 21
Derivation of the method
Starting from an initial guess: u0, we find the minimizers iteratively in the following order:
Θ0, u1,Θ1, . . . , uk,Θk, . . . .
Assuming that uk is calculated, we fix uk and find the minimizer of Eτ (uk,Θ) to obtain Θk:
part 1 : Θk = arg minΘ∈S
Eτ (uk,Θ)
and fix Θk to obtain uk+1:
part 2 : uk+1 = arg minu∈BEτ (u,Θk).
8/ 21
part1
Because Eτr is independent of Θ, one only needs to find the global minimizers of Ef withrespect to Θ to obtain Θk:
Θk = arg minΘ∈S
∫Ω
ukF1(f ,Θ) + (1− uk)F2(f ,Θ) dx
Since most of these models use strictly convex fidelity terms, each element Θi,j(i = 1, 2, j ∈ [m]) in Θk can be obtained via solving the following system of equations:
∂Ef
∂Θ1,1= 0, . . . ,
∂Ef
∂Θ1,m= 0,
∂Ef
∂Θ2,1= 0, . . . ,
∂Ef
∂Θ2,m= 0.
Above system can be approximately solved using the Gauss–Seidel strategy.I This step gives an optimal choice of Θ for the fixed uk!!!
9/ 21
part2I Eτ (u,Θk) : concave and quadratic.I The set B : = u ∈ BV(Ω,R) | u = 0, 1 contains the boundary points of the
following convex set K:
K =u ∈ BV(Ω,R) | u ∈ [0, 1]
(K is the convex hull of B).I The minimizer of a concave functional on a convex set can only be attained at the
boundary points of the convex set.I Relaxation:
uk+1 = arg minu∈KEτ (u,Θk)⇔ uk+1 = arg min
u∈BEτ (u,Θk)
I Linearization:
uk+1 = arg minu∈KLτ (f ,Θk, uk, u)
where Lτ (f ,Θk, uk, u) is the linearization of Eτ (u,Θk) at uk (to a constant),
Lτ (f ,Θk, uk, u) : =
∫Ω
uφ dx
and
φ = F1(f ,Θk)− F2(f ,Θk) + λ
√π
τGτ ∗ (1− 2uk).
10/ 21
Cont’d
I φ is explicitly computed using Θk and uk
I Because u(x) ∈ [0, 1], the minimization after linearization can be carried out in apointwise manner by checking whether φ(x) > 0 or not. That is, the minimum can beattained at
uk+1(x) =
1 if φ(x) ≤ 0,0 otherwise.
(1)
I Linearization⇒ Accelerating the convergence
11/ 21
The iterative convolution-thresholding method (ICTM)
Iterate the following three steps until convergence (no pixel switches from one segment to theother between two iterations)I For the fixed uk , find
Θk = arg minΘ∈S
∫Ω
ukF1(f ,Θ) + (1− uk)F2(f ,Θ) dx.
I Use Θk from Step 1 and evaluate
φk(x) = F1(f ,Θk)− F2(f ,Θk) + λ
√π
τGτ ∗ (1− 2uk).
I Set
uk+1(x) =
1 if φk(x) ≤ 0,0 otherwise.
Theorem (Stability1)Let (uk,Θk) be the k-th iteration derived in the ICTM. We have
Eτ (uk+1,Θk+1) ≤ Eτ (uk,Θk)
for any τ .
1Wang and Wang, 201912/ 21
Chan–Vese2 model
Θk = (Ck1,C
k2) Fi = |Ci − f |2
I For the fixed uk , compute
Ck1 =
∫Ω ukf dx∫Ω uk dx
, Ck2 =
∫Ω(1− uk)f dx∫
Ω 1− uk dx,
I Use Ck1 and Ck
2 from Step 1 and evaluate
φk(x) = F1(f ,Ck1)− F2(f ,Ck
2) + λ
√π
τGτ ∗ (1− 2uk).
I Set
uk+1(x) =
1 if φk(x) ≤ 0,0 otherwise.
2Chan and Vese, IEEE Trans. Image Process., 2001, Vese and Chan, Int. J. Comput. Vis., 200213/ 21
Number of iterations: 15. Parameters: (τ, λ) = (0.02, 0.05).
14/ 21
Locally Statistical Active Contour (LSAC) 3 model
Θi = (νi, b(x),Ci) Fi =∫Ω Iρ(x− y)
(log(νi) + |f (x)− b(y)Ci|2/2ν2
i
)dy
I For the fixed uk , compute
Ck1 =
∫Ω(Iρ ∗ bk−1)fuk dx∫Ω(Iρ ∗ bk−12)uk dx
,
Ck2 =
∫Ω(Iρ ∗ bk−1)f(1 − uk) dx∫Ω(Iρ ∗ bk−12)(1 − uk) dx
,
νk1 =
√√√√∫Ω
∫Ω Iρ(x − y)uk(x)(f(x) − bk−1(y)Ck
1)2 dydx∫Ω
∫Ω Iρ(x − y)uk(y) dydx
,
νk2 =
√√√√∫Ω
∫Ω Iρ(x − y)(1 − uk(x))(f(x) − bk−1(y)Ck
2)2 dydx∫Ω
∫Ω Iρ(x − y)(1 − uk(y)) dydx
,
bk(x) =[Ck
1/(νk1)2]Iρ ∗ (fuk) + [Ck
2/(νk2)2]Iρ ∗ (f(1 − uk))
[(Ck1/ν
k1)2]Iρ ∗ uk + [(Ck
2/νk2)2]Iρ ∗ (1 − uk)
.
I Use (νki , b
k(x),Cki ) to evaluate
φk(x) = F1 − F2 + λ
√π
τGτ ∗ (1− 2uk).
I Set
uk+1(x) =
1 if φk(x) ≤ 0,0 otherwise.
3Zhang et al., IEEE Trans. Cybernetics, 201615/ 21
# of iterations of the ICTM 8 7 7 7 7# of iterations of the level-set method [Zhang et al., 2016] 7 13 35 186 239
First row: Initial contour of the same image with different intensity inhomogeneity. Secondrow: The segmented region. Table: Comparison of the number of iterations for each case fromleft to right between the ICTM and the level-set method used in [Zhang et al. 2016]. In all fiveexperiments, we set ρ = 15, λ = 0.1, and τ = 0.01. 4
I JS index between two regions S1 and S2: JS(S1, S2) = |S1 ∩ S2|/|S1 ∪ S2|, whichdescribes the ratio between the intersection areas of S1 and S2. In the five experiments, wehave JS(S1, S2) = 1, 1, 0.9997, 0.9985, and 0.9985.
4The results for the level-set method are obtained using the software code fromhttps://www4.comp.polyu.edu.hk/˜cslzhang/LSACM/LSACM.htm
16/ 21
# of iterations of the ICTM 5 30 28 35 18# of iterations of the level-set method [Zhang et al., 2016] 57 219 670 290 230
17/ 21
Local Intensity Fitting (LIF)5 model
Θi = Ci(x) Fi(f ,Θ1,Θ2) = µi∫Ω Gσ(x− y)|Ci(x)− f (y)|2 dx (fixed µi)
I For the fixed uk , compute
Ck1(x) =
Gσ ∗ (ukf )Gσ ∗ uk + ε
, Ck2(x) =
Gσ ∗ ((1− uk)f )Gσ ∗ (1− uk) + ε
,
(small ε for the regularization)I Use Ck
1(x) and Ck2(x) from Step 1 and evaluate
φk(x) = F1(f ,Ck1(x))− F2(f ,Ck
2(x)) + λ
√π
τGτ ∗ (1− 2uk).
I Set
uk+1(x) =
1 if φk(x) ≤ 0,0 otherwise.
5Li et al., IEEE Trans. Image Process., 200818/ 21
# of iterations of the ICTM 15 25 43 28 47# of iterations of the level-set method [Li et al., 2008] - 256 131 117 209
Initial contour and segmented region using the ICTM in the LIF model 6.
6The results for the level-set method are obtained using the software code fromhttp://www.imagecomputing.org/˜cmli/code/.
19/ 21
The ICTM for n-segment caseUsing n characteristic functions:
ui(x) = χΩi (x) :=
1 if x ∈ Ωi,
0 otherwise,i ∈ [n].
Perimeter:
|∂Ωi| ≈√π
τ
n∑j=1,j 6=i
∫Ω
uiGτ ∗ uj dx.
I For the fixed uk , find
Θk = arg minΘ∈S
n∑i=1
∫Ω
uiFi(f ,Θ)dx.
I For i ∈ [n], evaluate
φki = Fi(f ,Θk) + 2λ
n∑j=1,j 6=i
√π
τGτ ∗ uk
j .
I For i ∈ [n], set
uk+1i (x) =
1 if i = minarg min`∈[n] φ
k`,
0 otherwise.
20/ 21
Conclusions
I We proposed a novel iterative convolution-thresholding method (ICTM) that is applicableto a range of models for image segmentation.
I Numerical experiments show that the method is simple, efficient, unconditionally stable,and insensitive to the number of segments.
I The ICTM converges in significant fewer iterations than the level-set method for all theexamples we tested.
Questions?
Email: [email protected] (Dong Wang), [email protected] (Xiao-Ping Wang)
References:I D. Wang and X.-P. Wang, The iterative convolution-thresholding method (ICTM) for
image segmentation, submitted, 2020.I D. Wang, H. Li, X. Wei, and X.-P. Wang, An efficient iterative thresholding method for
image segmentation, J. Comput. Phys., vol. 350, pp. 657–667, 2017.
21/ 21