Discrete versus Continuous sparse regularizationThe Deconvolution Problem Assumption:the signal m 0...

Discrete versus Continuous sparse regularization

Vincent Duval Gabriel Peyre

CNRS - Universite Paris-Dauphine

August 29, 2014

Vincent Duval, Gabriel Peyre Discrete vs continuous August 29, 2014 1 / 22

Outline

1 The deconvolution problem

2 Discrete regularization

3 Continuous regularization

Outline

Deconvolution

Measuring devices have a non sharp impulse response: our observationsare blurred of a ”true ideal scene”.

Geophysics,

Astronomy,

Microscopy,

Spectroscopy,

. . . Image courtesy of S. Ladjal

Goal: Obtain as much detail as we can from given measurements.

The Deconvolution Problem

Consider a signal m0 defined on T = R/Z (i.e. [0, 1) with periodicboundary condition).

Perturbation model:

Original Signal

Low-pass filter

t0.5−0.5

Observation

y0 + w

Goal: recover m0 from the observation y0 + w = ϕ ∗m0 + w (orsimply y0 = ϕ ∗m0)

Ill-posed problem:I the low pass filter might not be invertible (ϕn = 0 for some

frequency n)I even though, the problem is ill-conditioned (|ϕn| � |ϕ0| for high

frequencies n)

Consider a signal m0 defined on T = R/Z (i.e. [0, 1) with periodicboundary condition).

Perturbation model:

Original Signal

Low-pass filter

t0.5−0.5

Observation

y0 + w

Goal: recover m0 from the observation y0 + w = ϕ ∗m0 + w (orsimply y0 = ϕ ∗m0)Ill-posed problem:

I the low pass filter might not be invertible (ϕn = 0 for somefrequency n)

I even though, the problem is ill-conditioned (|ϕn| � |ϕ0| for highfrequencies n)

Assumption: the signal m0 is sparse

Original Signal

t0 1x1

Low-pass filter

t0.5−0.5

Observation

m0 =N∑i=1

aiδxi , where

ai ∈ R,xi ∈ T,N ∈ N is small.

so that we observe y + w =∑N

i=1 aiϕ(· − xi) + w.

Idea: Look for a sparse signal m such that ϕ ∗m ≈ y0 + w (or y0).

Question

How can we use sparsity to solve the inverse problem?

Can we guarantee that the reconstructed signal is close to the originalone?

What can we say about the structure/support of the solution?

Outline

Discretization

Define a finite grid G = { i2M

; 0 6 i 6 2M − 1} ⊂ T,

and consider signals of the form m =∑2M−1

i=0 aiδ i

Candidate Signal

2M−12M

ϕ ∗m =

2M−1∑i=0

aiϕ(· − i

ϕ ϕ(·− 12M

) . . . ϕ(· − 2M−12M

︸︷︷︸

aM−1

︸︷︷︸

Equivalent paradigm: Look for a sparse vector a ∈ R2M such thatΦa ≈ y0 (or Φa ≈ y0 + w).

Discrete `1 regularizationDefine

‖m‖`1(G) =

{ ∑2M−1i=0 |ai| if m =

∑2M−1i=0 aiδi/2M ,

+∞ otherwise.

m0 =∑2M−1

i=0 a0,iδi/2M

2M−12M

Basis Pursuit [Chen & Donoho (94)]

infm∈M(T)

‖m‖`1(G) such that Φm = y0 (PM0 (y0))

LASSO [Tibshirani (96)] or Basis Pursuit Denoising [Chen et al. (99)]

infm∈M(T)

λ‖m‖`1(G) +1

2||Φm− (y0 + w)||22 (PMλ (y0 + w))

`2-robustness ([Grasmair et al.(11)])

If m0 =∑i a0,iδi/2M is the unique solution to PM0 (y0), and mλ =

∑i aλ,i is a

solution to PMλ (y0 + w), then ‖aλ − a0‖2 = O(‖w‖2) for λ = C‖w‖2.

Discrete `1 regularizationDefine

‖m‖`1(G) =

{ ∑2M−1i=0 |ai| if m =

∑2M−1i=0 aiδi/2M ,

+∞ otherwise.

Basis Pursuit [Chen & Donoho (94)]

infm∈M(T)

‖m‖`1(G) such that Φm = y0 (PM0 (y0))

LASSO [Tibshirani (96)] or Basis Pursuit Denoising [Chen et al. (99)]

infm∈M(T)

λ‖m‖`1(G) +1

2||Φm− (y0 + w)||22 (PMλ (y0 + w))

`2-robustness ([Grasmair et al.(11)])

If m0 =∑i a0,iδi/2M is the unique solution to PM0 (y0), and mλ =

∑i aλ,i is a

solution to PMλ (y0 + w), then ‖aλ − a0‖2 = O(‖w‖2) for λ = C‖w‖2.

Robustness of the support (discrete problem)

Can one guarantee that Suppmλ = Suppm0?

Sufficient conditions for Suppmλ ⊆ Suppm0:I Exact Recovery Principle (ERC) [Tropp (06)]

I Weak Exact Recovery Principle (W-ERC) [Dossal & Mallat (05)]

Almost necessary and sufficient Suppmλ = Suppm0

I Fuchs criterion [Fuchs (04)]

Robustness of the support (discrete problem)

Can one guarantee that Suppmλ = Suppm0?

Sufficient conditions for Suppmλ ⊆ Suppm0:I Exact Recovery Principle (ERC) [Tropp (06)]

I Weak Exact Recovery Principle (W-ERC) [Dossal & Mallat (05)]

Almost necessary and sufficient Suppmλ = Suppm0

I Fuchs criterion [Fuchs (04)]

Optimality conditionsFrom convex analysis:

a0 is a solution to PM0 (y0) if and only if there exists p ∈ L2(T) such that thefunction η = Φ∗p satisfies:

∀i s.t. a0,i 6= 0, η

)= sign(a0,i) and ∀k,

∣∣∣∣η( k

)∣∣∣∣ 6 1;

aλ is a solution to PMλ (y0 + w) if and only if the functionηλ = 1

λΦ∗(y0 + w − Φ∗aλ) satisfies:

∀i s.t. aλ,i 6= 0, ηλ

)= sign(aλ,i) and ∀k,

∣∣∣∣ηλ( k

)∣∣∣∣ 6 1;

Fuchs theorem

For m0 =∑Ni=1 a0,iδx0,i , define

ηF = Φ∗pF , where

pF = argmin{‖p‖L2(T); (Φ∗p)(x0,i) = sign(a0,i)}

= Φ+,∗s.−1

Theorem ([Fuchs (04)])

Assume that {ϕ(· − x0,1), . . . ϕ(· − x0,N )} has full rank.If |ηF ( k

2M)| < 1 for all k such that k

2M/∈ {x0,1, . . . , x0,N}, then m0 is the unique

solution to PM0 (y0), and there exists α > 0, λ0 > 0 such that for 0 6 λ 6 λ0 and‖w‖2 6 αλ,

The solution mMλ to PMλ (y0 + w) is unique.

SuppmM = Suppm0, that is mMλ =

∑Ni=1 a

Mλ,iδx0,i , and sign(aλ,i) = sign(a0,i),

aMλ,I = a0,I + Φ+xIw − λ(Φ∗xIΦxI )−1 sign(a0,I).

If |ηF ( k2M

)| > 1 for some k, the support is not stable.

Fuchs theorem

For m0 =∑Ni=1 a0,iδx0,i , define

ηF = Φ∗pF , where

pF = argmin{‖p‖L2(T); (Φ∗p)(x0,i) = sign(a0,i)}

= Φ+,∗s.

Theorem ([Fuchs (04)])

Assume that {ϕ(· − x0,1), . . . ϕ(· − x0,N )} has full rank.If |ηF ( k

2M)| < 1 for all k such that k

2M/∈ {x0,1, . . . , x0,N}, then m0 is the unique

solution to PM0 (y0), and there exists α > 0, λ0 > 0 such that for 0 6 λ 6 λ0 and‖w‖2 6 αλ,

The solution mMλ to PMλ (y0 + w) is unique.

SuppmM = Suppm0, that is mMλ =

∑Ni=1 a

Mλ,iδx0,i , and sign(aλ,i) = sign(a0,i),

aMλ,I = a0,I + Φ+xIw − λ(Φ∗xIΦxI )−1 sign(a0,I).

If |ηF ( k2M

)| > 1 for some k, the support is not stable.

But. . .

When the grid is too thin, the Fuchs criterion cannot hold⇒ the support is not stable.

Theorem (D.-Peyre 2013)

If the Non Degenerate Source Condition holds, and M is large enough,the support at low noise contained in

⋃i∈I(i ∪ i+ εi) where εi = ±1 .

In general, the support at low noise is made at mostof pairs of consecutive spikes: the original one andone of its immediate neighbours.

2M−12M

Outline

Continuous setting (total variation)

As proposed in [de Castro & Gamboa (12), Candes & Fernandez-Granda (13), Bredies &

Pikkarainen (13), Tang et al. (12)], consider the

Total variation of measures:

|m|(T) = sup

{∫Tψdm;ψ ∈ C(T), ‖ψ‖∞ 6 1

∑Ni=1 a0,iδx0,i

Example : If m =∑Ni=0 aiδxi , then |m|(T) =

∑Ni=0 |ai|.

If m = fdL, then |m|(T) =∫T |f(t)|dt.

Rationale: the extreme points of {m ∈M(T), |m|(T) 6 1} are the Diracmasses: δx for x ∈ T.

Total variation regularization

Total variation of measures:|m|(T) = sup

{∫T ψdm;ψ ∈ C(T), ‖ψ‖∞ 6 1

}Basis Pursuit for measures [de Castro & Gamboa (12), Candes &

Fernandez-Granda (13)],

infm∈M(T)

|m|(T) such that Φm = y0 (P0(y0))

LASSO for measures [Bredies & Pikkarainen (13), Azais et al. (13)]

infm∈M(T)

λ|m|(T) +1

2||Φm− (y0 + w)||22 (Pλ(y0 + w))

Numerical methods for solving P0(y0) and Pλ(y0 + w) are proposed in [Bredies &

Pikkarainen (13), Candes & Fernandez-Granda (13)]

Identifiability for discrete measuresMinimum separation distance of a measure m:

∆(m) = minx,x′∈Suppm,x 6=x′

|x− x′|

Ideal Low Pass filter: ϕ(t) = sin(2fc+1)πt)sinπt

i.e ϕn = 1 for |n| 6 fc, 0 otherwise.

m =∑N

i=1 aiδxi

∆(m)

Theorem ((Candes and Fernandez-Granda, 2013))

Let ϕ be the ideal low-pass filter. There exists a constant C > 0 suchthat, for any (discrete) measure m0 with ∆(m0) > C

fc, m0 is the unique

solution of

infm∈M(T)

|m|(T) such that Φm = y0 (P0(y0))

where y0 = Φm0.

Remark: 1 6 C 6 1.87.

Robustness?

Weak-* robustness ([Bredies & Pikkarainen (13)])

If m0 =∑i a0,iδx0,i is the unique solution to P0(y0), mλ is a solution to

Pλ(y0 + w), then mλ∗⇀m0 as λ→ 0+, ‖w‖22/λ→ 0.

(see also [Azais et al. (13), Fernandez-Granda (13)] for robustness of local averages in the case of the ideal LPF)

Robustness?

Weak-* robustness ([Bredies & Pikkarainen (13)])

If m0 =∑i a0,iδx0,i is the unique solution to P0(y0), mλ is a solution to

Pλ(y0 + w), then mλ∗⇀m0 as λ→ 0+, ‖w‖22/λ→ 0.

(see also [Azais et al. (13), Fernandez-Granda (13)] for robustness of local averages in the case of the ideal LPF)

Optimality conditions

From convex analysis:

m0 =∑Ni=1 a0,iδx0,i is a solution to P0(y0) if and only if there exists p ∈ L2(T)

such that the function η = Φ∗p satisfies:

∀1 6 i 6 N, η (x0,i) = sign(a0,i) and ∀t ∈ T, |η (t)| 6 1;

mλ =∑N′

i=1 aλ,iδxλ,i is a solution to Pλ(y0 + w) if and only if the functionηλ = 1

λΦ∗(y0 + w − Φ∗aλ) satisfies:

∀1 6 i 6 N ′, ηλ (xλ,i) = sign(aλ,i) and ∀t ∈ T, |ηλ (t)| 6 1;

Robustness of the support (continuous problem)

For m0 =∑Ni=1 ai0δxi,0 , define

ηV = Φ∗pV , where

pV = argmin{‖p‖L2(T); (Φ∗p)(x0,i) = sign(ai,0),

and (Φ∗p)′(x0,i) = 0}−1

Robustness of the support (continuous problem)

For m0 =∑Ni=1 ai0δxi,0 , define

ηV = Φ∗pV , where

pV = argmin{‖p‖L2(T); (Φ∗p)(x0,i) = sign(ai,0),

and (Φ∗p)′(x0,i) = 0}

Theorem ((D.-Peyre 2013))

Assume that Γx0= (ϕ(· − x0,1), . . . ϕ(· − x0,N , ϕ′(· − x0,1), . . . ϕ′(· − x0,1)) has

full rank, and that m0 satisfies the Non Degenerate Source Condition, i.e.ηV = Φ∗pV satisfies

|ηV (t)| < 1 for all t ∈ T \ {x0,1, . . . , x0,N},η′′V (x0,i) 6= 0 for all 1 6 i 6 N .

Then there exists, α > 0, λ0 > 0 such that for 0 6 λ 6 λ0 and ‖w‖2 6 αλ,

the solution mλ to Pλ(y + w) is unique and has exactly N spikes,

mλ =∑Ni=1 aλ,iδxλ,i ,

the mapping (λ,w) 7→ (aλ, xλ) is C1.

Conclusion

Low noise stability analysis for the discrete and continuous frameworks

Imposing a grid leads to the apparition of parasitic spikes

Experiment yourself the Sparse Spikes Deconvolution on Numericaltours!

www.numerical-tours.com

Preprint:Exact Support Recovery for Sparse Spikes Deconvolution (2013),

V. Duval & G. Peyre,submitted to FoCM

Thank you for your attention!

Azais, J.-M., De Castro, Y., and Gamboa, F. (2013). Spike detection frominaccurate samplings. Technical report.

Bredies, K. and Pikkarainen, H. (2013). Inverse problems in spaces ofmeasures. ESAIM: Control, Optimisation and Calculus of Variations,19:190–218.

Candes, E. J. and Fernandez-Granda, C. (2013). Towards a mathematicaltheory of super-resolution. Communications on Pure and AppliedMathematics. To appear.

Chen, S. and Donoho, D. (1994). Basis pursuit. Technical report, StanfordUniversity.

Chen, S., Donoho, D., and Saunders, M. (1999). Atomic decomposition bybasis pursuit. SIAM journal on scientific computing, 20(1):33–61.

de Castro, Y. and Gamboa, F. (2012). Exact reconstruction using beurlingminimal extrapolation. Journal of Mathematical Analysis andApplications, 395(1):336–354.

Dossal, C. and Mallat, S. (2005). Sparse spike deconvolution withminimum scale. In Proceedings of SPARS, pages 123–126.

Fuchs, J. (2004). On sparse representations in arbitrary redundant bases.IEEE Transactions on Information Theory, 50(6):1341–1344.

Grasmair, M., Scherzer, O., and Haltmeier, M. (2011). Necessary andsufficient conditions for linear convergence of `1-regularization.Communications on Pure and Applied Mathematics, 64(2):161–182.

Tang, G., Bhaskar, B. N., and Recht, B. (2013). Near minimax linespectral estimation. CoRR, abs/1303.4348.

Tibshirani, R. (1996). Regression shrinkage and selection via the Lasso.Journal of the Royal Statistical Society. Series B. Methodological,58(1):267–288.

Tropp, J. (2006). Just relax: Convex programming methods for identifyingsparse signals in noise. IEEE Transactions on Information Theory,52(3):1030–1051.

Discrete versus Continuous sparse regularizationThe Deconvolution Problem Assumption:the signal m 0...

Documents

-sparse Reconstruction of Sharp Point Set Surfacesasharf/Projects/l1sparse-tog-1.15.pdf · 2.2 Sparse Signal Reconstruction Sparse signal processing has had over the last few years

Sparse deconvolution (an MM algorithm)...0 50 100 150 200 250 300-0.5 0 0.5 1 Sparse Signal 0 50 100 150 200 250 300-2-1 0 1 2 Convolution 0 50 100 150 200 250 300-1 0 1 2 Noisy data

Signal Processing Course : Theory for Sparse Recovery

A unified approach to sparse signal processing

Sparse Signal Processing Using iterative Method …ee.sharif.edu/~imat/imat_files/1.pdf19th International Conference on Telecommunications (ICT 2012) Sparse Signal Processing Using

Solving Sudoku by Sparse Signal Processing - DiVA portal792205/FULLTEXT01.pdf · Solving Sudoku by Sparse Signal Processing MUHAMMAD MOHSIN ABBASI Master’s Degree Project Stockholm,

Sparse Representations and the Basis Pursuit Algorithm - IEEE Signal Processing … · 2019-12-18 · Sparse Representation Theory. ... Research in Signal/Image Processing. Model

Recursive Recovery of Sparse Signal Sequences from ...namrata/TSP_RecRecon_Paper_revise_1.pdf · Recursive Recovery of Sparse Signal Sequences from Compressive Measurements: A Review

Sparse Signal Recovery: Theory, Applications and Algorithms

VLSI Architectures for Compressive Sensing and Sparse Signal

Comparison of Sparse Signal Separation Algorithms for Maritime … · 2018-08-17 · 1 Comparison of Sparse Signal Separation Algorithms for Maritime Radar Target Detection Brian

RFID Signal Reconstruction of Internet of Things Based on ... › proceedings_series...signal is sparse after sorting and exponentially decreasing and approaching zero. Sparse representation

Sparse Signal Recovery

A lava attack on the recovery of sums of dense and sparse ... · \sparse+dense" model, in which the signal is given by the sum of a sparse signal and a dense signal. Such a structure

Recursive Dynamic CS: Recursive Recovery of Sparse Signal ...home.engineering.iastate.edu/~namrata/TSP_RecRecon_Paper_4.pdf · Recursive Dynamic CS: Recursive Recovery of Sparse Signal

Group- sparse signal denoising: Non- convex Regularization ...ijrpublisher.com/gallery/70-december-2018.pdf · Group- sparse signal denoising: Non- convex Regularization, Convex Optimization

Sparse and Low Rank Representations in Music Signal Analysis

Symposium on Sparse Signal Processing and Deep … on Sparse Signal Processing and Deep ... The Sparse Signal Processing and Deep Learning symposium will explore deep connection

Discrete uncertainty principles and sparse signal processing · · 2017-05-26Discrete uncertainty principles and sparse signal processing ... digital signal processing has ... of

Sensing, Compression and Recovery for WSNs: Sparse Signal ... · 1 Sensing, Compression and Recovery for WSNs: Sparse Signal Modeling and Monitoring Framework Giorgio Quer, Member,