Connecting Bayesian and

Denoising-Based Approximate

Message

Chris Metzler, Richard Baraniuk

Rice University

Arian Maleki

Columbia University

Compressive Sensing Problem

Solution: Assume Structure

• Sparse: OMP [Tropp 04], IST [Figueiredo et al. 07], AMP [Donoho et al. 09]

• Minimal Total Variation: TVAL3 [Li et al. 09], TV-AMP [Donoho et al. 13]

• Tree-Sparse: Model-CoSaMP [Baraniuk et al. 10],Turbo-AMP [Som and Schniter 12]

• Group-Sparse: NLR-CS [Dong et al. 14]

Using Structure is Hard

• Write as penalty or constraint

• How to efficiently solve for non-convex R(x)?

• What is R(x) for natural images?

• What is R(x) for RF, microscopy, and other applications?

This Talk

• Develop algorithm that can easily use almost any structure

• Predict performance with accurate state evolution

• Derive theoretical guarantees• Measurements required

• Robust to noise

• Optimality and suboptimality

• Demonstrate state-of-the-art performance• 10dB better than wavelet sparsity

• 50x faster than group-sparsity

Insight: Denoisers Use Structure

• Gaussian Kernel• Smooth

• Soft Wavelet Thresholding [Donoho and Johnstone 94]

• Sparse wavelet representation

• BLS-GSM [Portilla et al. 03]

• Coefficients follow Gaussian Mixture Model

• NLM [Baudes et al. 05]

• Correlated structures

• BM3D [Dabov et al. 07]

• Group-sparse in DCT/Wavelet representation

• BM3D-SAPCA [Dabov et al. 09]

• Group-sparse in adaptive basis

Denoisers as Black Boxes

Denoiser

Denoisers as Projections

C

Naïve Algorithm: Denoising-based

Iterative Thresholding (D-IT)

Naïve Algorithm: D-IT

Our prior on x

Naïve Algorithm: D-IT

Naïve Algorithm: D-IT

Naïve Algorithm: D-IT

Naïve Algorithm: D-IT

Failure of D-IT: Systematic Errors

Systematic Errors: Overshooting

Systematic Errors: Overshooting

Too High

Systematic Errors: Overshooting

Too High Too Low

Too High Too Low

Systematic Errors: Overshooting

Too High Too Low

Too High Too Low

Systematic Errors: Overshooting

Too High Too Low

Too High Too Low

Systematic Errors: Overshooting

Too High Too Low

Too High Too Low

Use residuals from

previous iterations to

avoid over/under-

shooting

New Algorithm: D-AMP

Onsager Correction

D-AMP Benefits

D-IT

• Updates proportional to residual (P-controller).

• 5dB improvement over L1

D-AMP

• Updates proportional to previous residual (PI-controller).

• 10dB improvement over L1

(state-of-the-art)

• Onsager Correction

Onsager Correction:

• Where did it come from?• Approximation of message passing algorithm

Onsager Correction:

• Where did it come from?• Approximation of message passing algorithm

• Why does it help?• zt stores residuals over many iterations (momentum)

• Corrects for bias in denoiser solutions

• Makes errors uncorrelated (Gaussian) and thus easy to remove

Onsager Correction:

• Where did it come from?• Approximation of message passing algorithm

• Why does it help?• zt stores residuals over many iterations (momentum)

• Corrects for bias in denoiser solutions

• Makes errors uncorrelated (Gaussian) and thus easy to remove

• How is it calculated?• Approximation from Monte Carlo SURE [Ramani et al. 08]

D-AMP Avoids Systematic Errors

D-AMP Theoretical Properties

• State evolution predicts performance

• Explicit phase transition

• Robust to noise

• No algorithm can uniformly outperform D-AMP

• Single-class suboptimal

• Easy to tune

Bayesian-AMP and Bayesian

State Evolution

• Algorithm

• State Evolution

Denoising-based AMP and

Deterministic State Evolution

• Algorithm

• State Evolution

State Evolution Comparison

• With minimax denoiser, suprema of Deterministic and Bayesian state evolutions are equivalent:

• Significance of deterministic state evolution: • Can apply without knowing x’s distribution

• Can apply to natural images and other complex signals

State Evolution of D-IT and D-AMP

State Evolution is Accurate for

Many Denoisers

State Evolution for Discontinuous

Denoisers

State Evolution for Smoothed

Discontinuous Denoisers

Main Theoretical Results

• Denoiser Performance

• Phase Transition: Determined by denoiser

• Noise Sensitivity: Graceful failure

• No algorithm can uniformly outperform D-AMP

• D-AMP is single-class suboptimal

Parameter Tuning

• Denoiser parameters

• Problem: Tune denoiser parameters over multiple iterations

• Result: Greedy parameter selection is optimal

3x Under-Sampling

20x Under-Sampling

Wavelet Sparse (L1) BM3D-AMP (our algorithm)

10x Under-Sampling with Noise

NLR-CS BM3D-AMP

Performance without Noise

Computation Time

30x Faster

70x Faster

Performance with Noise

D-AMP Summary

• Arbitrary denoiser• NLM

• BM3D

• Useful state evolution

• State-of-the-art performance

• Resilient to noise

• >97% reduction in average computation time

C. Metzler, A. Maleki, R. G. Baraniuk, “From Denoising to

Compressed Sensing,” arXiv:1406.4175.pdf

D-AMP vs. AMP1, G-AMP

2, Turbo-

AMP3, TV-AMP

4, GrAMPA

5, etc.

Similarities

• Same basic AMP iterations

• Solve a series of denoisingproblems

• Better denoisers lead to better phase transition and noise sensitivity

Differences

• Separable denoisers without scale invariance

• Signal x can be denoised but need not have generalized-sparsity nor known px

• Approximate Onsager correction

• New deterministic state evolution

• State evolution holds for separable, but continuous, denoisers

• Derive phase transition and noise sensitivity of non-sparse signals

• Derive optimality/sub-optimality

• Optimal tuning strategy

1. Donoho et al. 09

2. Rangan 12

3. Som and Schniter 12

4. Donoho et al. 13

5. Borgerding et al. 14

Near Proper Denoiser

• Denoiser Performance

• Phase Transition: Determined by denoiser

• Noise Sensitivity: Graceful failure