Learned D-AMP: Principled Neural Network based Compressive …lcarin/Guoyin1.4.2019.pdf · 2019. 1. 4. · Learned D-AMP: Principled Neural Network based Compressive Image Recovery

Learned D-AMP: Principled Neural Network basedCompressive Image Recovery

Christopher A. Metzler, Ali Mousavi, Richard G. Baraniuk

Rice University

NIPS 2017

Presenter: Guoyin Wang

January 4, 2019

Christopher A. Metzler, Ali Mousavi, Richard G. Baraniuk (Rice)LD-AMP January 4, 2019 1 / 22

Overview

1 Major Contribution

2 ModelProblem DescriptionAlgorithmsTraining Network

3 Experiments


Major Contribution

Unrolling: a process to turn an iterative algorithm (D-AMP) into adeep neural net (LD-AMP) which is interpretable and maintainconvergence guarantees.Efficiently train a deep neural network.Outperform the state-of-the-art BM3D-AMP and NLR-CS algorithmsin terms of both accuracy and run time on compressive imagingproblem.


Compressive Imaging Problem

Linear measurements y ∈ Rm of the signal of interest x ∈ Rn iscaptured via y = Ax + ε, where A ∈ Rm×n is a measurement matrixand ε ∈ Rm is noise.Image Inverse Problems: given the measurements y and themeasurement matrix A, a computational imaging system seeks torecover x .Compressive Sensing (CS): when m < n this problem is ill-posedinverse problem, and prior knowledge about x must be used torecovery the signal, such as fact that x ∈ C , where C is the set of allnatural images.


Compressive Imaging Problem

When no measurement noise ε presented, we want to find the imagexo at the intersection of the set C and the affine subspace{x |y = Ax}The optimization formulation:

argminx‖y − Ax‖22 subject to x ∈ C . (1)


IT and AMP

Low cost iterative algorithm are proposed to solve this problem:

Iterative Thresholding (IT) Algorithm

IT Algorithmz t = y − Ax t ,

x t+1 = η(x t + AHz t).η(·) = Thresholding non-linearity (2)


IT and AMP

Effective noise νt = x t + AHz t − xo should be i.i.d Gaussian, but ITAlgorithm fails to maintain this property.

Approximate Message Passing (AMP) Algorithm

AMP Algorithm

bt = nz t−1〈η′(x t−1 + AHz t−1)〉m ,

z t = y − Ax t + bt ,

x t+1 = η(x t + AHz t). (3)

where 〈·〉 denotes the average of a vector, η′ represents the derivative of ηand b is Onsager correction term which removes the bias.


D-IT and D-AMP

Nonlinear function η can be replaced by a powerful denoiser.Say xo + σz is a noisy observation of a natural image, with xo ∈ C ,z ∼ N(0, I) and σ as the standard deviation of the noise.Denoiser Dσ would simply find the point in the set C that is closestto the observation xo + σz

Dσ(xo + σz) = argminx‖xo + σz − x‖22 subject to x ∈ C . (4)

Such denoiser is a projection onto C , ideally should return estimate xcloser to xo than xo + σz .


D-IT and D-AMP

Denoising-based IT (D-IT) Algorithm

D-IT Algorithm z t = y − Ax t ,

x t+1 = Dσ̂t (x t + AHz t). (5)

where σ̂t is the estimated standard deviation of effective noise νt . Notethat

Dσ̂t (x t + AHz t) = Dσ̂t (xo + νt) (6)


D-IT and D-AMP

Denoising-based AMP (D-AMP) Algorithm

D-AMP Algorithmbt = z t−1divDσ̂t−1(x t−1 + AHz t−1)

m ,

z t = y − Ax t + bt ,

σ̂t = ‖z t‖2√m ,

x t+1 = Dσ̂t (x t + AHz t). (7)

‖zt‖2√m serves as a useful and accurate estimate of the standard deviation of

νt and typically, D-AMP algorithms use a Monte-Carlo approximation forthe divergence divD(·).


D-IT and D-AMP

(a) D-IT Iterations (b) D-AMP Iterations

Figure: Reconstruction behavior of D-IT (left) and D-AMP (right) with anidealized denoiser. Because D-IT allows bias to build up over iterations of thealgorithm, its denoiser becomes ineffective at projecting onto the set C of allnatural images. The Onsager correction term enables D-AMP to avoid this issue.


Unrolling Process

Example: IT Algorithm:

IT Algorithmz t = y − Ax t ,

x t+1 = η(x t + AHz t).η(·) = Thresholding non-linearity


Unrolling Process

z t = y − Ax t ,

x t+1 = η(x t + AHz t).z t+1 = y − Ax t+1,

x t+2 = η(x t+1 + AHz t+1).

Feed training data, i.e., (xo, yo) pairs as label-input pairs, feedsforward through the network, calculate errors and backpropagate.The free parameters of η are leaned during training.


Unroll D-AMP

Unroll D-AMP

bl =z l−1divDl

σ̂l−1(x l−1 + AHz l−1)m ,

z l = y − Ax l + bl ,

σ̂l = ‖z l‖2√m ,

x l+1 = Dlσ̂l (x l + AHz l). (8)


CNN-based Denoiser

A denoiser that easily propagates gradients is needed.Denoising Convolutional Neural Network (DnCNN) satisfies this andis more accurate and far faster than competing techniques like BM3D.

Consists of 16 to 20 CNN layers.First layer uses 64 3× 3× c filters (c is number of color channels).Next 14 to 18 layers use 64 3× 3× 64 filters + batch normalization.Final layer uses c 3× 64 filters to reconstruct.


Learned D-AMP

LDAMP Neural Network

bl =z l−1divDl

w l−1(σ̂l−1)(xl−1 + AHz l−1)

m ,

z l = y − Ax l + bl ,

σ̂l = ‖z l‖2√m ,

x l+1 = Dlw l (σ̂l )(x

l + AHz l). (9)

where Dlw l (σ̂l ) to indicate that layer l of the network uses denoiser Dl , that

this denoiser depends on its weights/biases w l , and that these weightsmay be a function of the estimated standard deviation of the noise σ̂l .


Leaned D-AMP

Figure: Two layers of the LDAMP neural network. When used with the DnCNNdenoiser, each denoiser block is a 16 to 20 convolutional-layer neural network.

LD-IT network is nearly identical but does not compute Onsagercorrection term.


Training LDIT and LDAMP

End-to-end training: Train all the weights of the networksimultaneously.Layer-by-layer training: Train a 1 AMP layer network to recover thesignal, fix these weights, add an AMP layer, and repeat until we havetrained a 10 layer network.Denoiser-by-denoiser training: Decouple the denoisers from therest of the network and train each on denoising problems at differentnoise levels.Later-by-layer training and denoiser-by-denoiser training for LDAMPare found minimum-mean-squared-error (MMSE) optimal.


Experiments

Datasets Berkeley’s BSD-500 dataset, 400 images for training, 50 forvalidation, and 50 for testing.Evaluation Metric PSNR = 10 log10( 2552

mean((x̂−xo)2)) when the pixelrange is 0 to 255.


Results

(a) Original Image (b) TVAL3 (26.4 dB, 6.85 sec) (c) BM3D-AMP (27.2 dB,75.04 sec)

(d) LDAMP (28.1 dB, 1.22sec)

Figure: Reconstructions of 512× 512 Boat test image sampled at a rate ofmn = 0.05 using coded diffraction pattern measurements and no measurementnoise. LDAMP’s reconstructions are noticeably cleaner and far faster than thecompeting methods.


Results

Table: PSNRs and run times (sec) of 128× 128 reconstructions withi.i.d. Gaussian measurements and no measurement noise at various sampling rates.

Methodmn = 0.10 m

n = 0.15 mn = 0.20 m

n = 0.25

PSNR Time PSNR Time PSNR Time PSNR Time

TVAL3 21.5 2.2 22.8 2.9 24.0 3.6 25.0 4.3BM3D-AMP 23.1 4.8 25.1 4.4 26.6 4.2 27.9 4.1LDIT 20.1 0.3 20.7 0.4 21.1 0.4 21.7 0.5LDAMP 23.7 0.4 25.7 0.5 27.2 0.5 28.5 0.6NLR-CS 23.2 85.9 25.2 104.0 26.8 124.4 28.2 146.3


Results

Table: PSNRs and run times (sec) of 128× 128 reconstructions with codeddiffraction measurements and no measurement noise at various sampling rates.

Methodmn = 0.10 m

n = 0.15 mn = 0.20 m

n = 0.25

PSNR Time PSNR Time PSNR Time PSNR Time

TVAL3 24.0 0.52 26.0 0.46 27.9 0.43 29.7 0.41BM3D-AMP 23.8 4.55 25.7 4.29 27.5 3.67 29.1 3.40LDIT 22.9 0.14 25.6 0.14 27.4 0.14 28.9 0.14LDAMP 25.3 0.26 27.4 0.26 28.9 0.27 30.5 0.26NLR-CS 21.6 87.82 22.8 87.43 25.1 87.18 26.4 86.87


Documents

Learned D-AMP: Principled Neural Network based Compressive …lcarin/Guoyin1.4.2019.pdf · 2019. 1. 4. · Learned D-AMP: Principled Neural Network based Compressive Image Recovery