A Transform-based Variational Framework

Guy Gilboa

Pixel Club, November, 2013

In a Nutshell

Spatial Input

Transform Analysis

Transform Filtering

Spatial Output

𝐼Φ→𝑆𝐻

→𝑆𝐻Φ

−1

→𝐼

Fourier inspiration:

Fourier Scale Fourier Scale

𝐹→

𝐿𝑃𝐹→

𝐹−1

→

Spectral

TV Flow

0 20 40 60 80 100 120 1400

500

1000

1500

2000

2500

3000

TV ScaleTV Scale

0 20 40 60 80 100 120 1400

500

1000

1500

2000

2500

3000

Relations to eigenvalue problemsGeneral linear: (L linear operator)Functional based

uLu

)div( uuu || 21 dxuJH

|| dxuJTV ||

div uu

u

Linear

Nonlinear

What can a transform-based approach give us?Scale analysis based on the

spectrum.New types of filtering – otherwise

hard to design: nonlinear LPF, BPF, HPF.

Nonlinear spectral theory – relation to eigenfunctions and eigenvalues.

Deeper understanding of the regularization, optimal design with respect to data, noise and artifacts.

Examples of spectral applications today:Eigenfunctions for 3D processing

Taken from Zhang et al, “Spectral mesh processing”, 2010.

Taken from L Cai, F Da, “Nonrigid deformation recovery..”, 2012.

Image Segmentatoin

Eigenvectors of the graph Laplacian[Taken from I. Tziakos et al, “Color image segmentation using Laplacian eigenmaps”, 2009 ]

Some Related StudiesAndreu, Caselles, Belletini, Novaga et al 2001-

2012– TV flow theory.Steidl et al 2004 – Wavelet – TV relationBrox-Weickert 2006 – scale through TV-flowLuo-Aujol-Gousseau 2009 – local scale measuresBenning-Burger 2012 – ground states (nonlinear

spectral theory)Szlam-Bresson – Cheeger cuts.Meyer, Vese, Osher, Aujol, Chambolle, G.

and many more – structure-texture decomposition.Chambolle-Pock 2011, Goldstein-Osher 2009 –

numerics.

Scale Space – a Natural Way to Define Scale

We’ll talk specifically about total-variation (TV-flow, Andreu et al - 2001):

)( ,| , 0 uJpfupu utt

xxfxun

u

Du

Du

t

u

in ),();0(

),0(on ,0

),0(in ,||

div

Scale space as a gradient descent:

TV-Flow:A behavior of a disk in time[Andreu-Caselles et al–2001,2002, Bellettini-Caselles-Novaga-2002, Meyer-2001]

Center of disk, first and second time derivatives:

t

… …

𝑢 𝑢𝑡 𝑢𝑡𝑡2

0 −0 .5

0

0

Spectral TV basic framework

Phi(t) definition

txtxt utt );();(

Reconstruction

Reconstruction formula

Th. 1: The reconstruction formula recovers

fdttf

0

)(ˆ

dxxff )(||

1

Spectral response

Spectrum S(t) as a function of time t:

dxxtL

xttS |);(|);()( 1

t

0 20 40 60 80 100 120 1400

500

1000

1500

2000

2500

3000

S(t)f

Spectrum example

0 5 10 15 20 25 30 35 40 45 500

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

2x 10

4

tS

(t)

f S(t)

Dominant scales

0 5 10 15 20 25 30 35 40 45 500

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

2x 10

4

t

S(t

)

);2( xt );10( xt );37( xt

Eigenvalue problem

The nonlinear eigenvalue problem with respect to a functional J(u) is defined by:

We’ll show a connection to the spectral components .

),(

,

uJp

up

)(t

Solution of eigenfunctions

Th. 2: For is an eigenfunction with eigenvalue then:

1)()

1()(

)()1

();(

1

,0

10 ),1)((

);(

LxfttS

xftxt

t

ttxfxtu

What are the TV eigenfunctions?In 2D, is a characteristic function of a convex set. I then is an eigenfunction.

Area

Perimeterboundary)on curvaturemax(

𝜅 (𝑝 )=1𝑟;𝑃 (𝐶 )|𝐶|

=2𝜋𝑟𝜋𝑟2

=2𝑟 [Giusti-1978], [Finn-1979],[Alter-

Caselles-Chambolle-2003].

Filtering

Let H(t) be a real-valued function of t. The filtered spectral response is

)();(:);( tHxtxtH

fdtxtxf HH

0

);()(

The filtered spatial response is

𝜙(𝑡) 𝜙𝐻 (𝑡)H(t)

Filtering, example 1:

TV Band-Pass and Band-Stop filters

Band-pass Band-stop0 2 4 6 8 10 12 14

0

0.5

1

1.5

2

2.5x 10

4

f S(t)

Disk band-pass example

0 20 40 60 80 100 120 1400

500

1000

1500

2000

2500

3000

S(t)

We have the basic framework

Spatial Input

Transform Analysis

Transform Filtering

Spatial Output

𝐼Φ→𝑆𝐻

→𝑆𝐻Φ

−1

→𝐼

txtxt utt );();(

LxttS 1);()(

0 20 40 60 80 100 120 1400

500

1000

1500

2000

2500

3000

fdtxtxf HH

0

);()(

Numerics Many ways to solve.Variational approach was chosen:

Currently use Chambolle’s projection algorithm (some spikes using Split-Bregman, under investigation).

In time: ◦ 2nd derivative - central difference◦ 1st derivative - forward differnce◦ Discrete reconstruction algorithm proved for

any regularizing scale-space (Th. 4).

0))1(()()1( nutpnunu ||)(||2

1)(

2

2nuut

uJL

)(uput

TV-Flow as a LPFTh. 3: The solution of the TV-flow is equivalent to spectral filtering with:

0 10 20 30 40 50 60 70 80 90 1000

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

t

HT

FV

,t1

t1 = 1t1 = 5t1 = 10t1 = 20

,

0 ,0

11

1

, 1

ttt

tt

ttH tTVF

Nonlocal TVReminder: NL-TV (G.-Osher

2008):

Gradient

Functional

),()()()( yxwxuyuxuw

dxxuuJ wTVNL |)(|)(

yx,

Spectral NL-TV?The framework can fit in principle

many scale-spaces, like NL-TV flow. We can obtain a one-homogeneous regularizer.

What is a generalized nonlocal disk?What are possible eigenfunctions? It is expected to be able to process

better repetitive textures and structures.

Sparseness in the TV senseSparse spectrum – the signal

has only a few dominant scales.

Or many small ones (here TV energy is large)

Can be a large objects

Natural images – are not very sparse in general

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 50

500

1000

1500

2000

2500

S(t)

t

0 5 10 15 20 25 30 35 40 45 500

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

2x 10

4

t

S(t

)

0 20 40 60 80 100 120 1400

500

1000

1500

2000

2500

3000

Noise Spectrum

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 50

200

400

600

800

1000

1200

1400

S(t)

t

𝑆 (𝑡)→

Various standard deviations:

S(t)

Noise + signal

Not additive. Spreads original image spectrum. Needs to be investigated.

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 50

500

1000

1500

2000

2500

Clean image

Noise only

Image with noiseuf

f-u

Band-pass filtered

0 5 10 15 20 250

2000

4000

6000

8000

10000

12000

t

Signal

Noise

Signal + Noise

Spectral Beltrami Flow?Initial trials on Beltrami flow with parameterization such that it is closer to TVOriginal Beltrami Flow Spectral

Beltrami

Difference images:

• Keeps sharp contrast

• Breaks extremum principle

Values along one line (Green channel)

0 20 40 60 80 100 1200.4

0.5

0.6

0.7

0.8

0.9

1

1.1

Original

Spectral Beltrami

Beltrami Flow

Spectral Beltrami

Segmentation priorsSwoboda-Schnorr 2013 –

convex segmentation with histogram priors.

We can have 2D spectrum with histograms

Use it to improve segmentation

S(t,h)

Texture processingMany texture bands

We can filter and manipulate certain bands and reconstruct a new image.

Generalization of structure-texture decomposition.

t

tdtt

i

i

1

)(Band(i) 1

,..2,1

ii tt

i

Processing approachDeconstruct the image into bands

Identify salient textures

Amplify / attenuate / spatial process the bands.

Reconstruct image with processed bands

Color formulation

Vectorial TV – all definitions can be generalized in a straightforward manner to vector-valued images.

Bresson-Chan (2008) definition and projection algorithm is used for the numerics.

Orange example

Orange – close up

Original Modes 2,3=0 Modes 2-5=x1.5

Selected phi(t) modes (1, 5, 15, 40)

residual

f

Old man

Old man – close up

Original 2 modes attenuated 7 modes attenuated

Old Man - First 3 Modes

Modes: 1 2 3

Take Home Messages Introduction of a new TV

transform and TV spectrum. Alternative way to understand

and visualize scales in the image.

Highly selective scale separation, good for processing textures.

Can be generalized to other functionals.

Thanks!

Refs. Google “Guy Gilboa publications”• Preliminary ideas are in SSVM 2013 paper. • Most material is in CCIT Tech report 803.• Up-to-date and organized - submitted journal

version – contact me.