The iterative convolution-thresholding method (ICTM) for imagesegmentation

Dong Wang (University of Utah)Xiao-Ping Wang (HKUST)

1/ 21

Introduction

I Image: f : Ω→ [0, 1]d

I Ω : domain of image (discrete pixels)I d : number of channels (e.g., gray image: d = 1, color image: d = 3.)

I Image segmentation: the process of partitioning a digital image into multiple segmentsm

find the optimal partition in the sense of minimizing a objective functional

2/ 21

Objective functionals

Intuitively, the optimal partition should obey:I In each segment, the feature should be “almost constant” or has no “big” jumps. The

partition should be consistent with the image itself.I The boundary of the partition should be “relatively smooth” (try to ignore some noise from

in the image).

Mathematically, in the objective function, we should haveI A quantity to measure the consistency of the image in each segment (Fidelity term)I A quantity to measure the smoothness of the boundary of each segment (Regularization

term)

Generally, for the n-segment (Ω1,Ω2, · · · ,Ωn) case, we write the general objective functionalinto

E =n∑

i=1

∫Ωi

Fi(f ,Θ1, . . . ,Θn) dx +n∑

i=1

λi|∂Ωi|

I Fi: choice of Fi depends on the features of the image or the types of imageI Θi: possible parameters or functions needed to describe Fi

I λi: constant parametersI |∂Ωi|: the perimeter of the segment Ωi

3/ 21

Example

In each segment, require the feature of the image is close to a constant

⇓Fi(f ,Ci) = |Ci − f |2 (Ci: average of f in the segment Ωi)

⇓Objective functional

E(Ω1, . . . ,Ωn,C1, . . . ,Cn) =∑n

i=1∫Ωi|Ci − f |2 dx + λi

∑ni=1 |∂Ωi| (Chan–Vese Model)

[Chan and Vese, IEEE Trans. Image Process., 2001, Vese and Chan, Int. J. Comput. Vis., 2002]

4/ 21

Representing the partition and expressing the objective functionalExisting numerical method:I Prime dual, split Bregmann, framelets, K-means clustering, · · ·I Phase-field based methodI Level set method

Level set methodThe discrete image is interpreted as a function defined on a continuous domain.

⇓The boundary of the segment is represented as the levelset of an auxiliary function ϕ.

⇓Every term in the objective function can be expressed using ϕ and the Heaviside function of ϕ.

⇓Variation w.r.t ϕ gives a partial differential equation (PDE)

⇓Solving the PDE to evolve the initial guess to the final result to get the segmentation

Benefits: Very easy to adapt to arbitrary modelsDisadvantages:I Solving complicated PDEsI Reinitialization (or another technique) is needed for the auxiliary function ϕI Small artificial time step for the stability reason (converges slow)I Sensitive to numbers of segments

5/ 21

Using characteristic functions (2-segment case)I The first segment Ω1 is denoted by its characteristic function u(x), i.e.,

u(x) :=

1 if x ∈ Ω1,

0 otherwise.

I the characteristic function of the second segment Ωc1 is 1− u(x).

I the interface between two segments is now implicitly represented by u(x).

I The first term in the objective function∫Ω1

F1 dx +∫Ω2

F2 dx =∫Ω uF1 + (1− u)F2 dx

I The perimeter term [Miranda et al., Annales de la faculte des sciences de ToulouseMathematiques, 2007]

|∂Ω1| ≈√π

τ

∫Ω

uGτ ∗ (1− u) dx

6/ 21

Approximation of the objective functional

E ≈ Eτ (u,Θ): = Ef (u,Θ) + Eτr (u,Θ)

I Ef (u,Θ) =∫Ω uF1(f ,Θ) + (1− u)F2(f ,Θ) dx

I Eτr (u,Θ) = λ√πτ

∫Ω uGτ ∗ (1− u) dx

Target: Find uτ,? such that

(uτ,?,Θτ,?) = arg minu∈B,Θ∈S

Eτ (u,Θ)

where B : = u ∈ BV(Ω,R) | u = 0, 1, S is the admissible set of Θ, and BV(Ω,R)denotes the bounded-variation functional space.

7/ 21

Derivation of the method

Starting from an initial guess: u0, we find the minimizers iteratively in the following order:

Θ0, u1,Θ1, . . . , uk,Θk, . . . .

Assuming that uk is calculated, we fix uk and find the minimizer of Eτ (uk,Θ) to obtain Θk:

part 1 : Θk = arg minΘ∈S

Eτ (uk,Θ)

and fix Θk to obtain uk+1:

part 2 : uk+1 = arg minu∈BEτ (u,Θk).

8/ 21

part1

Because Eτr is independent of Θ, one only needs to find the global minimizers of Ef withrespect to Θ to obtain Θk:

Θk = arg minΘ∈S

∫Ω

ukF1(f ,Θ) + (1− uk)F2(f ,Θ) dx

Since most of these models use strictly convex fidelity terms, each element Θi,j(i = 1, 2, j ∈ [m]) in Θk can be obtained via solving the following system of equations:

∂Ef

∂Θ1,1= 0, . . . ,

∂Ef

∂Θ1,m= 0,

∂Ef

∂Θ2,1= 0, . . . ,

∂Ef

∂Θ2,m= 0.

Above system can be approximately solved using the Gauss–Seidel strategy.I This step gives an optimal choice of Θ for the fixed uk!!!

9/ 21

part2I Eτ (u,Θk) : concave and quadratic.I The set B : = u ∈ BV(Ω,R) | u = 0, 1 contains the boundary points of the

following convex set K:

K =u ∈ BV(Ω,R) | u ∈ [0, 1]

(K is the convex hull of B).I The minimizer of a concave functional on a convex set can only be attained at the

boundary points of the convex set.I Relaxation:

uk+1 = arg minu∈KEτ (u,Θk)⇔ uk+1 = arg min

u∈BEτ (u,Θk)

I Linearization:

uk+1 = arg minu∈KLτ (f ,Θk, uk, u)

where Lτ (f ,Θk, uk, u) is the linearization of Eτ (u,Θk) at uk (to a constant),

Lτ (f ,Θk, uk, u) : =

∫Ω

uφ dx

and

φ = F1(f ,Θk)− F2(f ,Θk) + λ

√π

τGτ ∗ (1− 2uk).

10/ 21

Cont’d

I φ is explicitly computed using Θk and uk

I Because u(x) ∈ [0, 1], the minimization after linearization can be carried out in apointwise manner by checking whether φ(x) > 0 or not. That is, the minimum can beattained at

uk+1(x) =

1 if φ(x) ≤ 0,0 otherwise.

(1)

I Linearization⇒ Accelerating the convergence

11/ 21

The iterative convolution-thresholding method (ICTM)

Iterate the following three steps until convergence (no pixel switches from one segment to theother between two iterations)I For the fixed uk , find

Θk = arg minΘ∈S

∫Ω

ukF1(f ,Θ) + (1− uk)F2(f ,Θ) dx.

I Use Θk from Step 1 and evaluate

φk(x) = F1(f ,Θk)− F2(f ,Θk) + λ

√π

τGτ ∗ (1− 2uk).

I Set

uk+1(x) =

1 if φk(x) ≤ 0,0 otherwise.

Theorem (Stability1)Let (uk,Θk) be the k-th iteration derived in the ICTM. We have

Eτ (uk+1,Θk+1) ≤ Eτ (uk,Θk)

for any τ .

1Wang and Wang, 201912/ 21

Chan–Vese2 model

Θk = (Ck1,C

k2) Fi = |Ci − f |2

I For the fixed uk , compute

Ck1 =

∫Ω ukf dx∫Ω uk dx

, Ck2 =

∫Ω(1− uk)f dx∫

Ω 1− uk dx,

I Use Ck1 and Ck

2 from Step 1 and evaluate

φk(x) = F1(f ,Ck1)− F2(f ,Ck

2) + λ

√π

τGτ ∗ (1− 2uk).

I Set

uk+1(x) =

1 if φk(x) ≤ 0,0 otherwise.

2Chan and Vese, IEEE Trans. Image Process., 2001, Vese and Chan, Int. J. Comput. Vis., 200213/ 21

Number of iterations: 15. Parameters: (τ, λ) = (0.02, 0.05).

14/ 21

Locally Statistical Active Contour (LSAC) 3 model

Θi = (νi, b(x),Ci) Fi =∫Ω Iρ(x− y)

(log(νi) + |f (x)− b(y)Ci|2/2ν2

i

)dy

I For the fixed uk , compute

Ck1 =

∫Ω(Iρ ∗ bk−1)fuk dx∫Ω(Iρ ∗ bk−12)uk dx

,

Ck2 =

∫Ω(Iρ ∗ bk−1)f(1 − uk) dx∫Ω(Iρ ∗ bk−12)(1 − uk) dx

,

νk1 =

√√√√∫Ω

∫Ω Iρ(x − y)uk(x)(f(x) − bk−1(y)Ck

1)2 dydx∫Ω

∫Ω Iρ(x − y)uk(y) dydx

,

νk2 =

√√√√∫Ω

∫Ω Iρ(x − y)(1 − uk(x))(f(x) − bk−1(y)Ck

2)2 dydx∫Ω

∫Ω Iρ(x − y)(1 − uk(y)) dydx

,

bk(x) =[Ck

1/(νk1)2]Iρ ∗ (fuk) + [Ck

2/(νk2)2]Iρ ∗ (f(1 − uk))

[(Ck1/ν

k1)2]Iρ ∗ uk + [(Ck

2/νk2)2]Iρ ∗ (1 − uk)

.

I Use (νki , b

k(x),Cki ) to evaluate

φk(x) = F1 − F2 + λ

√π

τGτ ∗ (1− 2uk).

I Set

uk+1(x) =

1 if φk(x) ≤ 0,0 otherwise.

3Zhang et al., IEEE Trans. Cybernetics, 201615/ 21

# of iterations of the ICTM 8 7 7 7 7# of iterations of the level-set method [Zhang et al., 2016] 7 13 35 186 239

First row: Initial contour of the same image with different intensity inhomogeneity. Secondrow: The segmented region. Table: Comparison of the number of iterations for each case fromleft to right between the ICTM and the level-set method used in [Zhang et al. 2016]. In all fiveexperiments, we set ρ = 15, λ = 0.1, and τ = 0.01. 4

I JS index between two regions S1 and S2: JS(S1, S2) = |S1 ∩ S2|/|S1 ∪ S2|, whichdescribes the ratio between the intersection areas of S1 and S2. In the five experiments, wehave JS(S1, S2) = 1, 1, 0.9997, 0.9985, and 0.9985.

4The results for the level-set method are obtained using the software code fromhttps://www4.comp.polyu.edu.hk/˜cslzhang/LSACM/LSACM.htm

16/ 21

# of iterations of the ICTM 5 30 28 35 18# of iterations of the level-set method [Zhang et al., 2016] 57 219 670 290 230

17/ 21

Local Intensity Fitting (LIF)5 model

Θi = Ci(x) Fi(f ,Θ1,Θ2) = µi∫Ω Gσ(x− y)|Ci(x)− f (y)|2 dx (fixed µi)

I For the fixed uk , compute

Ck1(x) =

Gσ ∗ (ukf )Gσ ∗ uk + ε

, Ck2(x) =

Gσ ∗ ((1− uk)f )Gσ ∗ (1− uk) + ε

,

(small ε for the regularization)I Use Ck

1(x) and Ck2(x) from Step 1 and evaluate

φk(x) = F1(f ,Ck1(x))− F2(f ,Ck

2(x)) + λ

√π

τGτ ∗ (1− 2uk).

I Set

uk+1(x) =

1 if φk(x) ≤ 0,0 otherwise.

5Li et al., IEEE Trans. Image Process., 200818/ 21

# of iterations of the ICTM 15 25 43 28 47# of iterations of the level-set method [Li et al., 2008] - 256 131 117 209

Initial contour and segmented region using the ICTM in the LIF model 6.

6The results for the level-set method are obtained using the software code fromhttp://www.imagecomputing.org/˜cmli/code/.

19/ 21

The ICTM for n-segment caseUsing n characteristic functions:

ui(x) = χΩi (x) :=

1 if x ∈ Ωi,

0 otherwise,i ∈ [n].

Perimeter:

|∂Ωi| ≈√π

τ

n∑j=1,j 6=i

∫Ω

uiGτ ∗ uj dx.

I For the fixed uk , find

Θk = arg minΘ∈S

n∑i=1

∫Ω

uiFi(f ,Θ)dx.

I For i ∈ [n], evaluate

φki = Fi(f ,Θk) + 2λ

n∑j=1,j 6=i

√π

τGτ ∗ uk

j .

I For i ∈ [n], set

uk+1i (x) =

1 if i = minarg min`∈[n] φ

k`,

0 otherwise.

20/ 21

Conclusions

I We proposed a novel iterative convolution-thresholding method (ICTM) that is applicableto a range of models for image segmentation.

I Numerical experiments show that the method is simple, efficient, unconditionally stable,and insensitive to the number of segments.

I The ICTM converges in significant fewer iterations than the level-set method for all theexamples we tested.

Questions?

Email: dwang@math.utah.edu (Dong Wang), mawang@ust.hk (Xiao-Ping Wang)

References:I D. Wang and X.-P. Wang, The iterative convolution-thresholding method (ICTM) for

image segmentation, submitted, 2020.I D. Wang, H. Li, X. Wei, and X.-P. Wang, An efficient iterative thresholding method for

image segmentation, J. Comput. Phys., vol. 350, pp. 657–667, 2017.

21/ 21