Image Processing and Computational Mathematics

Tony F. Chan Hong Kong University of Science and Technology

Talk prepared in collaboration with

Xavier Bresson, Department of Computer Science, City University of Hong Kong

60th SIAM Annual Meeting

July 10, 2012

Input also from Ernie Esser, UC Irvine and Luminita Vese, UCLA.

(SIAM founded ? 1952. TC since January 1952.)

-  Relatively new to SIAM Community -  SIIMS founded 2007. Now 2nd highest impact journal in

all Applied Math (out of 245). #1 in Imaging Science SIIMS proposal

-  Imaging journals: SIAM SIIMS IEEE TIP JMIV IJCV

-  Imaging conferences: SIAM Imaging conference Scale-Space conference IEEE ICIP

- The new “CFD”. Part of “ Data Science”

2

Imaging science

3

4

IEEE TRANSACTIONS ON IMAGE PROCESSING A PUBLICATION OF THE IEEE SIGNAL PROCESSING SOCIETY Volume 7, Number 3, March 1998

4

What SIAM brings to IP

5

Image Processing and Applied Maths

Math IP

PDEs (CFD, viscosity solution) Calculus of variations

Differential geometry (level set method) Smooth & non-smooth functional analysis

(wavelets/Besov, total variation/BV)

Compressed sensing, L1 optimization (Bregman) Convexification for geometric problems

Relaxation for graph optimization problems

6

Timeline

1985 1990 2000 2005 2010 1995

Mumford-Shah segmentation

Perona-Malik AD Rudin-Osher-Fatemi

TVL2

Denoising/Segmentation Chan-Shen

Chan-Shen-Zhou TVWI

Candes-Romberg-Tao Donoho (CS)

Inpainting/CS

Kass-Witkin-Terzopoulos Snake

Caselles-Kimmel-Sapiro Geodesic Active Contour

Chan-Vese ACWE

Vese-Chan multiphase

Segmentation

Chan-Esedoglu TVL1

Chan-Esedoglu-Nikolova CMS Bresson-Esedoglu-Osher-etal

Geometric convexification

Rudin-Osher-Fatemi Gradient Flow

Osher-Burger-Goldfarb-Xu-Yin Bregman

Goldstein-Osher Split-Bregman

TV Optimization

IEEE special issue SIAG IS SIAM SIIMS

SIAM IS Conf Scale Space Conf

IP journals/confs

Bertalmio-Sapiro- Caselles-Ballester

Chan-Golub-Mulet Newton

7

IP challenges

-  Design non-linear models to handle image discontinuities (edges)

-  Make the most of geometry

-  Develop fast algorithms to deal with important data volumes

(RGB, time, multispectral components)

[Ron Kimmel]

edges

8

PDEs in Image Processing

-  Scale space filtering [Witkin 1983] -  Optimal approximations by piecewise smooth functions and

associated variational problems [Mumford, Shah 1989] -  Scale-space and edge detection using anisotropic diffusion [Perona,

Malik 1990] -  Motion of level sets by mean curvature motion [Evan, Spruck 1991] -  Nonlinear total variation based noise removal algorithms [Rudin,

Osher, Fatemi 1992] -  Axioms and fundamental equations of image processing [Alvarez,

Guichard, Lions, Morel 1993] -  Feature-oriented image enhancement using shock filters [Osher,

Rudin 1990]

9

Perona & Malik ‘90

Model properties: -  Remove noise while preserving edges -  Introduce feedback: adapt diffusity g

to evolving image u(x,t)

Nonlinear diffusion equation: ∂tu = div(g(|∇u|2)∇u)

where •  is a fuzzy edge detector •  diffusivity g is e.g. |∇u|2

g(|∇u|2) = 1

1 + |∇u|2/λ2

Existence of unique smooth solution (using Gaussian regularization in g): [Catte, Lions, Morel, Coll 1992]

10

Rudin, Osher & Fatemi ‘92 Variational model: min

uTV (u) +

α

2Ku− u02

-  Allows for edge capturing (discontinuities along level curves) -  TVD schemes popular for shock capturing -  TV controls both size of discontinuity and geometry of boundaries

Gradient flow: ∂tu = div(∇u

|∇u| )− (KKu−Ku0)∂u

∂n= 0

anisotropic diffusion data fidelity

11

Inpainting

Unifying TV restoration and inpainting processes:

minu

E∪D|∇u|+ λ

2

E|u− u0|2

λ =

−div(∇u

|∇u| ) + λ(u− u0) = 0

λE x ∈ E

0 x ∈ D

From Geary Gallery

12

Not reliable in non-repeating

regions.

Original !Image!Texture !

Synthesis!Elastica !

Inpainting!

Combining Texture Synthesis (Efros & Leung 97) & Elastica Inpainting (C, Kang, Shen 2001) (C, Ni, Roble SIGGRAPH 2007)

13

Original! Result!

image provided by Doug Enright!

14

PDEs & differential geometry in IP

-  Level set method: Fronts propagating with curvature-dependent speed: algorithms based on Hamilton-Jacobi formulations [Osher & Sethian ‘88]

-  Image segmentation: [Mumford & Shah ‘89] [Chan & Vese ‘01] [Vese & Chan ‘02] [Chung & Vese ’03] [Lie, Lysaker & Tai ‘06]

15

Mumford & Shah ‘89 Mumford-Shah optimization problem:

mins,C

Ω(s− u0)

2 + λ

Ω\C|∇s|2 + γ|C|

Conjecture [MS ‘89] (still open): There exists (at least) one minimizer s.t. 1) 2) C is made up of a finite union of -regular arcs, with some special cases: crack tip and triple junction (where three branches meet with angle)

s ∈ C1(Ω \ C)C1

120o

Original image Discontinuity part (edges) Smooth part

[Hewer, Kenney & Manjunath]

Chan & Vese ‘01 - Two-phase piecewise constant Mumford-Shah model:

minC,c1,c2

γ|C|+

inside(C)|u0 − c1|2 +

outside(C)|u0 − c2|2

data fidelity geometric regularization (length)

- Powerful technique: level set method [Osher & Sethian ‘88]

Inside C

Outside C

Outside C 0<!

0>!

0>!

0=!C

n!n!

n!

!!"

#$$%

&

'

'=

'

'=

|| ,

|| Normal

((

(( divKCurvaturen!

0),(|),( == yxyxC !

boundary of an open domain

Variational formulation and level set method [Zhao, Chan, Merriman and Osher ’96]

The Heaviside function

The level set formulation of the active contour model:

!"#

<

$=

0 if ,00 if ,1

)(%

%%H

The Euler-Lagrange equations:

Length: |C| =

Ω|∇H(φ)|

Area (inside(C)): |C| =

ΩH(φ)

minφ,c1,c2

γ

Ω|∇H(φ)|+

Ω|u0 − c1|2H(φ) +

Ω|u0 − c2|2(1−H(φ))

∂tφ = δε(φ)γdiv(

∇φ

|∇φ| )− |u0 − c1|2 + |u0 − c2|2

Europe Galaxy

18

Extension

- Image segmentation using a multilayer level-set approach [Chung & Vese ‘03] Ωi = x : li ≤ φ(x) ≤ li+1

Ωi = x : φ(x) = li

- A variant of the level set method and applications to image segmentation [Lie, Lysaker & Tai ‘06]

- A multiphase level set framework for image segmentation using the Mumford and Shah model [Vese & Chan ‘02] n level set functions → 2n regions

19

Geometric harmonic analysis in IP

-  JPEG (DCT), JPEG2000 (DWT) -  Wavelets [Mallat ’89, Coifman, Meyer, Wickerhauser ‘92] -  Curvelets [Starck, Candes, Donoho ‘02], bandelets [Le

Pennec, Mallat ’05] -  Denoising by soft-thresholding [Donoho ‘95] -  Besov spaces and wavelet shrinkage [Chambolle, De Vore,

Lee, Lucier ‘98] -  ENO wavelets [Chan, Zhou ‘03]

-  Will not elaborate in this talk.

20

BV, TV analysis in IP

-  Image Recovery via Total Variation Minimization and Related Problems [Chambolle, Lions 1997]

-  Functions of Bounded Variation and Free Discontinuity Problems [Ambrosio, Fusco, Pallara 2000]

-  Oscillating patterns in image processing and nonlinear evolution equations [Meyer 2001]

-  TV-L1: Aspects of total variation regularized L1 function approximation [Chan, Esedo!lu 2005]

-  The discontinuity set of solutions of the TV denoising problem and some extensions [Caselles, Chambolle, Novaga 2007]

21

Structure-texture decomposition [Meyer ‘01]

- [Rudin, Osher, Fatemi ‘92] decomposes image f into u in BV and v in L2: min

(u,v)∈BV×L2:f=u+v

|∇u|+ λv22

- [Meyer ‘01] decomposes f into u in BV and v in the dual space G: min

(u,v)∈BV×G:f=u+v

|∇u|+ λvG

Norm of G:

- Numerical implementation [Vese & Osher ‘03, Aujol & Chambolle ‘05]

ROF Meyer

vG := inf gL∞ : v = div(g), g = (g1, g2), g1, g2 ∈ L∞, |g(x)| =|g1|2 + |g2|2(x)

=> a signal with large oscillations has a small G-norm

22

TV-L1 [Chan-Esedoglu ’05] - [Rudin, Osher, Fatemi ‘92] preserves edges but loses contrast [Strong-Chan ‘96, Bellettini-Caselles-Novaga ’02].

TV-L1: min(u,v)∈BV×L1:f=u+v

|∇u|+ λv1

Model properties: - robust to contrast and geometry perturbation in the presence of noise - does not perturb a clean image in the absence of noise - Cleaner image multiscale decomposition than ROF - Data driven scale selection (detection of meaningful objects in images)

See also [Tadmor-Nezzar-Vese ‘03]

Theorem: If u0 = 1D, D is convex and smooth then

u =1− |∂D|

2λ|D| Contrast Lost

1D

23

Scale-Space Generated with ROF and TV-L1

Multiscale Decomposition with TV-L1

ROF

TV-L1 1

λ

24

Image Processing and Applied Maths

Math IP

PDEs (CFD, viscosity solution) Calculus of variations

Differential geometry (level set method) Smooth & non-smooth functional analysis

(wavelets/Besov, total variation/BV)

Compressed sensing, L1 optimization (Bregman) Convexification for geometric problems

Relaxation for graph optimization problems

25

TV Wavelet inpainting [Chan, Shen & Zhou ‘04]

Original image downloaded from internet Damaged image: 50% of wavelet coefficients (including low frequencies) are randomly lost.

Can you recognize this person?

•  Model I (for noise-less images):

No parameters. Problem dimension << #coefs.

•  Model II (for noisy images):

minβj,k

|∇u| s.t. u =

j,k

βj,kΦj,k, βj,k = β0j,k (observed coefs)

minβj,k

|∇u|+ λIu− u02 (u0 observed image)

wavelets

26

Restoration result

Original image Damaged image, 50% of wavelet coefficients are randomly lost. PSNR = 10.9

Model II: Recovered image, keeping undamaged coefficients unchanged. PSNR 18.8.

27

PSNR vs %Coefs Retained X-axis: percentage of randomly retained wavelet coefficients Y-axis: improvements in PSNR after inpainted by Model I (green) and Model II (red). As less coefficients are missed, the improvement in PSNR becomes larger.

An Extreme Example Upper left: Original clear square. Upper right: Lost all but one nonzero coefficients in the low-low frequency subband, while keep all high frequencies (PSNR=11.2dB). Lower left: Recovered by Model I. Perfect reconstruction (PSNR=61dB). Lower Right: Cross-section at x=128. The inpainted profile is visually indistinguishable from the original. A preview of Compressed Sensing!

28

Compressed Sensing in IP [Candes, Romberg & Tao ‘06, Donoho ’06]

Reconstruct the signal u in Rn from m measurements f with m<<n :

Assume u is sparse then the problem is:

Theorem: For m>=c.log n, the signal u can be reconstructed exactly (under reasonable assumptions) solving the problem:

minu

Ψu0 s.t. f = Au

L1 relaxation (tight) minu

Ψu1 s.t. f = Au

Ψu1 = TV (u) =

|∇u|

=> The CS problem is equivalent to the TV wavelet inpaiting problem.

(system is underdetermined) f = Au, u ∈ Rn, f ∈ Rm, A ∈ Rn×m

For A = RΦand ": sensing basis (wavelet, Fourier) R: measurement extractor

29

Reconstruction

CS reconstruction of an MR image using 30% of the k-space data [Goldstein & Osher ‘09]

CS reconstruction of an MR image using 30% of the Fourier space data [Zhang, Burger, Bresson & Osher ‘10]

30

Convexification of geometric problems in IP

-  Basic IP problems (segmentation, registration, surface reconstruction) are defined as non-convex optimization problems.

-  Curse of non-convexity: local minimizers/dependence of initialization to capture meaningful solution

-  Convexification: change non-convex geometric problem into convex relaxation problem which admits tight or exact relaxation [Strang ‘83, Chan, Esedoglu & Nikolova, ‘06, Bresson, Esedoglu, Vandergheynst, Thiran, Osher ’07]

31

TV-L1 and shape denoising Shape denoising (non-convex) TV-L1 (convex)

minΣ

Per(Σ) + λ|Σ∆S|

Ω|∇u|+

Ω|u− u0| u0 = 1S

Theorem: For any minimizer u* of TV-L1 then the thresholded function is a global minimizer of the shape denoising problem.

1u>µ

Proof: TV L1(u) =

µ

Per(u(x) > µ

Σ(µ)

) + |Σ(µ)∆S|dµ

Coarea formula Layer cake formula

We have the same geometric problem for each level set of u

Equivalence between convex problem (min over function in R) and non-convex problem (min over geometric sets)

32

Image segmentation

Level set method (non-convex):

minφ,c1,c2

γ|∇Hε(φ)|+

ΩHε(φ)|u0 − c1|2 +

Ω(1−Hε(φ))|u0 − c2|2

Why non-convexity is not desirable? Because some local minimizers are not meaningful:

Segmentation (non-convex) [Chan, Vese ’01]:

minΣ,c1,c2

γPer(Σ) +

Σ|u0 − c1|2 +

Ω\Σ|u0 − c2|2

33

Convexification [Chan, Esedoglu, Nikolova ’06, Strang ‘83]

minu,c1,c2

Fseg = γ

Ω|∇u|+

Ω|u0 − c1|2u+

Ω|u0 − c2|2(1− u)

Theorem: For any minimizer u* of Fseg then the thresholded function is a global minimizer of the segmentation problem.

1u>µ

Proof:

Coarea formula Layer cake formula

We have the same geometric problem for each level set of u

Fseg(u) =

µ

Per(Σ(µ)) +

Σ(µ)|u0 − c1|2 +

Ω\Σ(µ)|u0 − c2|2

dµ

Full convex relaxation of [Chan & Vese ‘01] w.r.t. [Brown-Chan-Bresson ‘12]:

(Σ, c1, c2)

34

Generalization -  Growing literature to reformulate most fundamental non-convex IP

problems as convex problems. No theoretical guarantee for exact relaxation but numerics show tight approximate solutions => the new level set method ?

-  Registration (optical flow-based registration [Horn & Schunck ‘81])

-  Surface Reconstruction [Savadjiev, Ferrie & Siddiqi ’03]

-  Multi-class segmentation [Potts ‘52]

minΣ

Per(Σ)−

∂Σ< NΣ, Np >

minΣi1≤i≤n

n

i=1

Perwp(Σi) + Areawia(Σi)

∪ni=1Σi = Ω, Σi ∩ Σj = ∅ ∀i = j

minu

Ω|∇u|+

Ω|I1 − I2(u)|2

35

L1 relaxation of graph cut problems Some graph cut problems are geometric problems => use L1 relaxation ideas to design better graph algorithms.

Data clustering can be cast as graph cut problems [Cheeger ‘70]:

minΩ

Cut(Ω,Ωc)

min(|Ω|, |Ωc|)

Balanced cut problems are NP-hard ⇒ continuous relaxations are required:

1) L2 relaxation [Shi, Malik ‘00]:

(Rayleigh quotient)

but L2 relaxation is not tight:

2) L1 relaxation [Szlam-Bresson ’10, Chung ’98]:

=> exact relaxation but non-convex optimization problem

minf :V→R

fH1

||f −mean(f)||22, fH1 =

i,j

wi,j |fi − fj |2

minf :V→R

fTV

f −median(f)1, fTV =

i,j

wi,j |fi − fj |

Cut(Ω,Ωc) =

i∈Ω, j∈Ωc

wi,jwith

36

Unsupervised and transductive learning problems

- Unsupervised clustering of high-dimensional data [Szlam-Bresson ‘10]

- Transductive clustering [Bresson, Tai, Chan & Szlam‘12]

MNIST: Clustering of Digit Numbers 70,000 28 # 28 images (dim=784)

Error of classification: -  [Shi & Malik]: 19.1% -  [Szlam & Bresson]: 13.1%

minΩ1,...,ΩK

K

k=1

cut(Ωk,Ωck)

min(|Ωk|, |Ωck|)

∪Kk=1Ωk = V and Ωi ∩ Ωj = ∅ ∀i = j

[2] Belkin, Problems of learning on manifolds, 2003

=> L1 works significantly better than L2 for small sets of labels.

and given labels lkKk=1

37

Non-smooth/TV optimization in IP -  Nonlinear total variation based noise removal algorithms[Rudin,

Osher, Fatemi 1992]: discretized PDEs -  A nonlinear primal-dual method for total variation-based image

restoration [Chan, Golub, Mulet 1996]: primal-dual algorithm -  An algorithm for total variation minimization and applications

[Chambolle 2004]: dual algorithm -  Smooth minimization of non-smooth functions [Nesterov 2005]: -  The split Bregman method for L1 regularized problems [Goldstein,

Osher 2009]: ADMM for L1 problems - A general framework for a class of first order primal-dual algorithms

for convex optimization in imaging science [Esser, Zhang, Chan 2009]

-  A first-order primal-dual algorithm for convex problems with applications to imaging [Chambolle, Pock 2011]: accelerated primal-dual algorithm

38

L1 Minimization and Convex Optimization

-  Many IP problems are formulated as L1 problems:

Basis Pursuit:

TV Denoising:

Geometric Convexification:

minu

u1 such that Au = f

minu

∇u1 +λ

2u− f2

-  Challenges: Such problems are large scale, nondifferentiable but convex and separable.

-  Renewed interest in classical convex optimization methods for L1 problems: new applications, improved algorithms and new theoretical developments

minu

∇u1 + λ < u,w(x) >

39

Algorithm Connections, Old and New Augmented Lagrangian/ Method of Multipliers

Bregman Iteration (Adds back the noise! Very effective for L1 minimization problems) [Osher, Burger, Goldfarb, Xu, Yin 2005]

Example: Basis Pursuit minu

u1 such that Au = f

↔

Alternating Direction Method of Multipliers (ADMM); Douglas Rachford Splitting

Split Bregman (Effective for TV minimization problems and much more generally applicable) [Goldstein, Osher 2008]

Example: TV Denoising minu

∇u1 +λ

2u− f2

↔

uk+1 = argminu

µu1 +1

2Au− fk2

fk+1 = f + (fk −Auk+1)

minu,z

z1 +λ

2u− f2 s.t. z = ∇u

↔uk+1 = argminu

µu1+

< pk, f −Au > +1

2Au− f2

pk+1 = pk + (f −Auk+1)

(uk+1, zk+1) = argminu,z

z1 +λ

2u− f2

+ pk, z −∇u+ δ

2∇u− z2

pk+1 = pk + δ(zk+1 −∇uk+1)

zk+1 = argminz

z1 +δ

2z −∇uk +

pk

δ2

uk+1 = argminu

λ

2u− f2 + δ

2∇u− zk+1 − pk

δ2

pk+1 = pk + δ(zk+1 −∇uk+1)

40

Algorithm Connections, Old and New

Uzawa’s Method Linearized Bregman (Efficient for large scale problems) [Darbon, Osher, Yin, Goldfarb 2007]

Proximal Forward Backward Splitting Fixed Point Continuation (FPC) and many related iterative thresholding methods for L1 minimization [Hale, Yin, Zhang 2007]

Arrow-Hurwicz Primal Dual Hybrid Gradient (PDHG) (Excellent for TV denoising) [Zhu, Chan 2008]

Preconditioned Method of Multipliers/ Proximal Method of Multipliers

Bregman Operator Splitting (BOS) (Proposed for nonlocal TV; very generally applicable) [Zhang, Burger, Bresson, Osher 2009]

↔↔↔↔

Quadratic Penalty Penalty method (Reformulates total variation penalty as constrained optimization problem) [Wang, Yin, Zhang 2007] ↔

Preconditioned ADMM (Linearization of penalty also related to surrogate function and optimization transfer)

Split Inexact Uzawa, Modified PDHG, Chambolle/Pock Method, He/Yuan Variants... (Many versions of this very versatile method) [Zhang, Burger, Osher, Esser, Chan, Chambolle, Pock...2009]

↔

Newton-like Methods CGM [Chan, Golub & Mulet '95] Semismooth Newton for TV [Hintermuller, Stadler] (Uses second order information, can be superlinearly convergent)

↔

41