Image denoising with patch estimation and low patch-rank regularization

Multimed Tools Appl (2014) 71:485–495DOI 10.1007/s11042-013-1535-4

Image denoising with patch estimation and lowpatch-rank regularization

Bo Li ·Ge Lin ·Qiang Chen ·Hongyi Wang

Published online: 28 August 2013© Springer Science+Business Media New York 2013

Abstract In this paper, we propose an image denoising algorithm for one specialclass of images which have periodical textures and contaminated by poisson noiseusing patch estimation and low patch-rank regularization. In order to form thedata fidelity term, we take the patch-based poisson likelihood, which will effectivelyremove the ‘blurring’ effect. For the sparse prior, we use the low patch-rank as theregularization, avoiding the choosing of dictionary. Putting together the data fidelityand the prior terms, the denoising problem is formulated as the minimization ofa maximum likehood objective functional involving three terms: the data fidelityterm; a sparsity prior term, in the form of the low patch-rank regularization ;and anon-negativity constraint (as Poisson data are positive by definition). Experimental

This paper is supported by the National natural science foundation of China(No. 61262050,61232011), NSFC-Guangdong Joint Fund (No. U1201252, U1135003),Foundation of Jiangxi Educational Committee (No. GJJ12441), Natural ScienceFoundation of Jiangxi (20122BAB211003).

B. LiCollege of Mathematics and Information Science,Nanchang Hangkong University, Nanchang 330063, Chinae-mail: [email protected]

G. Lin (B)National Engineering Research Center of Digital Life,State-Province Joint Laboratory of Digital Home Interactive Applications,School of Information Science and Technology, Sun Yat-sen University,Guangzhou 510006, Chinae-mail: [email protected]

Q. Chen (B)Guangdong University of Education, Guangzhou, Chinae-mail: [email protected]

H. WangCollege of Mathematics and Information Science, Nanchang Hangkong University,Nanchang 330063, China

486 Multimed Tools Appl (2014) 71:485–495

results show that the new method performs well for this special class of images whichhave periodical texture, and even for images with not strictly periodical textures.

Keywords Patch estimation ·Low patch-rank ·Proximal splitting method

1 Introduction

One of the most important task in computer vision is image denoising. The generalidea is to regard a noisy image f as being obtained by corrupting a noiseless imageu; given a model for the noise corruption, the desired image u is the solution of thecorresponding inverse problem.

Although, the image denoising problem is the simplest possible inverse problem,it’s still a difficult task for researchers due to the complicated structures and texturesof natural images. There exists no algorithm perform well for all images. So in thispaper, we focus on one special class of images which have periodical textures andcontaminated by poisson noise. The motivation for considering this class of imagesis that many scene in our real living environments is regular and symmetric, e.g.buildings,fabric,etc.(such as shown in Fig. 1). And this class of images can encompassa much richer class of textures. As we will see, our method works equally well whenthe structure to be reconstructed is not strictly periodic.

Generally, the degraded imaging process for additive noise can be describedmathematically by the first class Fridman operator equation

(Af )(x, y) =∫ ∫

K(x − ξ, y − η)m(x, y)dxdy + noise = d(x, y)

where K is usually a linear kernel function. For deblurring problem, it’s the pointspread function(psf), and for denoising, it’s just the identity operator. Since the theinverse problem is extremely ill-posed, most denoising procedures have to employsome sort of regularization. Generally, the image denoising problem can be solvedunder the frame of Tikhonov regularization.

minm

J (m) = ||Km − d||2 + α�[m]

Fig. 1 Natural images with periodical textures, the left is buildings, and right is fabric image

Multimed Tools Appl (2014) 71:485–495 487

where �[m] is the stability functional for m. According to different types of problems,the �[m] can adopt different regularizations. A very successful algorithm is that ofRudin et al. [15], Rudin and Osher [16], which uses total-variation regularization.The ROF model regards u as the solution to a variational problem, to minimize thefunctional

F(m) = ||Km − d||2 + λ

∫�

|∇m|

The TV regularization have two fundamental properties: firstly, it tends to preserveedge locations, and under certain conditions it preserves edge locations exactly;secondly, the intensity change is exactly inversely proportional to local featurescale,this is why TV image restoration can remove smaller-scaled noise, while leavinglarger-scaled features essentially intact. See [10] for an explanation of this in thecontext of Bayesian statistics.

For images with low-rank structures, the rank regularization is a good choice. Butthe key problem is that the rank of an image matrix is difficult to be formulated inclosed form. Recently, [20] showed that under surprisingly broad conditions, one canexactly recover the low-rank matrix A from D = A + E with gross but sparse errorsE by solving the following convex optimization problem:

arg minA

||D − A||2 + ||A||∗

Here,|| · ||∗ represents the nuclear norm of a matrix (the sum of its singular values).In [20], this optimization is dubbed Robust PCA (RPCA), because it enables one tocorrectly recover underlying low-rank structure in the data, even in the presenceof gross errors or outlying observations. This optimization can be easily recast asa semidefinite program and solved by an off-the-shelf interior point solver [2].However, although interior point methods offer superior convergence rates, thecomplexity of computing the step direction is O(m6). So they do not scale well withthe size of the matrix. In recent years, many scalable algorithms for high-dimensionalconvex optimization problems has been prompted, see references [8, 19, 22].

Many images, however, contain noise that satisfies a Poisson distribution. Themagnitude of Poisson noise varies across the image, as it depends on the imageintensity. This makes removing such noise very difficult. A familiar example isthat of radiography. The signal in a radiograph is determined by photon countingstatistics and is often described as particle-limited, emphasizing the quantized andnon-Gaussian nature of the signal. Removing noise of this type is a more difficultproblem. Besbeas et al. [1] review and demonstrate wavelet shrinkage methodsfrom the now classical method of Donoho [5] to Bayesian methods of Kolaczyk[13] and Timmermann and Novak [18]. These methods rely on the assumption thatthe underlying intensity function is accurately described by relatively few waveletexpansion coefficients. Kervrann and Trubuil [12] employ an adaptive windowingapproach that assumes locally piecewise constant intensity of constant noise variance.The method also performs well at discontinuity preservation. Jonsson et al. [11] usetotal variation to regularize positron emission tomography in the presence of Poissonnoise, and use a fidelity term similar to what we use below. Other methods, such assparse-inducing regularizations have also been proposed in [7, 21]. In 2011, Dupeet al. [6] propose an image deconvolution algorithm under poisson noise, which


is based on the statistical model. It’s a pointwise parameter estimation problem,while achieving good results for many images, it also suffers from the ‘pointwise’processing, and results in ‘blurring’ effects. Moreover, how to choose the properdictionary is also a difficult problem.

Inspired by the work of Dupe et al. [6], and in order to improve the per-formance of the cases analyzed above, in this paper, we propose an image de-noising algorithm for one special class of images which have periodical texturesand contaminated by poisson noise using patch estimation and low patch-rankregularization. Our main contribution includes: (1) Compared with the pointwiseparameter estimation in the work [6], the proposed method expand it to the multi-dimensional poisson distribution, and adopt the 2d patch-wise estimation and allowthe overlapping of patches which effectively reduce the ‘blurring’ effect; (2) Onedifficult problem in processing methods based on the dictionary learning is thechoosing of dictionary, in this paper, we adopt the patch-rank as the regulariza-tion, which guarantees the property of sparsity while avoiding the definition ofdictionary.

The paper is organized as follows. We state the problem to be solved in Section 2.The new denoising algorithm is proposed and analyzed in Section 3, and thenumerical experiments are given in Section 4. The final section is conclusion.

2 Problem statement

We consider the image formation model where an input image of n pixels x iscontaminated by Poisson noise. For the observation y, it should admit the followingpoisson PDF

y = P(xi)

where P(xi) is the poisson PDF with parameter xi, xi is the estimation of expection.Based on the above assumption, the Max-Likehood function can be represented

L(x) =n∏

i=1

P(yi) =n∏

i=1

P(yi|xi) =n∏

i=1

(xi)yi e−xi

yi!

the negative log-likehood function is

− ln L(x) = − lnn∏

i=1

(xi)yi e−xi

yi! = −n∑

i=1

yi ln(xi) + xi − ln yi!

In [6] the objective is to minimize the negative likehood function with someregularization and non-negative constraints

x = minx

− ln L(x) + φ(x) + ιC


The f irst term − ln L(x) is data-fidelity function, it’s an appropriate likelihoodfunction, obtained by the distribution of the observed data y given an original x,and it reflects the Poisson statistics of the noise. However, because it uses pixel-by-pixel max-likehood estimation as the data-f idelity term, the reconstructed image willbe ‘blurred’.

The second term φ(x) is a regularization for x, as the image is supposed to be eco-nomically (sparsely) represented in a pre-chosen dictionary D, so the regularizationterm φ(x) is a sparsity-inducing penalty. The dif f icult problem is how to choose theproper dictionary.

ιC is a positive constraints, since we are fitting Poisson intensities, which arepositive by nature.

In order to improve the performance of the cases analyzed above, in this paper,we propose a new algorithm based on patch estimation and low patch-rank regu-larization. In order to form the data fidelity term, we take the patch-based poissonlikelihood, which will effectively remove the ‘blurring’ effect. For the sparse prior,we use the low patch-rank as the regularization, avoiding the choosing of dictionary.Putting together the data fidelity, the sparse prior terms and the non-negativityconstraint (as Poisson data are positive by definition), we obtain our final denoisingmodel, and we establish the well-posedness of our optimization problems and givesome numerical experiments.

3 Proposed denoising algorithm using patch estimation and low patch-rankregularization

As the observation image y is a poisson random variable, more precisely, y is theobservation of the poisson variable, and for an m × n image, the observation yi,i = 1, ..., mn, should be Independent Identity Distribution(IID). According to theproperty of the distribution of Poisson, the joint of several IID variable should alsoobey poisson distribution.

Based on the above knowledge, we propose the patch-based estimation model.Firstly the image will be divided into small patches Patch(x) with size p1 × p2(asshown in Fig. 2). As each element of the patch is IID, So the distribution of eachpatch is

Pp1 p2(λ) = λ∑n

j=1 yi e−nλ

y1!y2!...yn!

where n is the number of elements in each patch, so n = p1 × p2. y1, y2, ..., yn isthe observation of each element, and the parameter λ is the expection, here we canuse the value of central pixel xi to approximate it, so the likelihood function can bedescribed as

L(xi) =∏y∈Cy

Py = x∑p1×p2

j=1 y j

i e−nxi

y1!y2!...yn!

where Cy is the observation of each patch with size p1 × p2.


Fig. 2 Patch map. Each patchis written as a r2 vector, andfor images with periodicaltextures the overall collectionof these vectors should havelow rank

The negative log-likelihood function is

− ln L(xi) =∑y∈Cy

p1×p2∑j=1

−y j ln xi + nxi (1)

The above log-likelihood function is the new data-f idelity term.For the sparse prior, we adopt the low patch-rank as the regularization. It has two

advantages, firstly, it’s not constructive and doesn’t need to choose the dictionarymanually, it’s self-learning; secondly, as the sparsity prior, the patch-rank reflects thestructural of image more effective than L1 norm. The patch-rank is firstly proposedby Schaeffer and Osher [17], they assume that the each image should be made upby some base patches, and expect the number of base textures to be low, since animage may exhibit only a few individual patterns. Therefore, the overall collectionof patches can be spanned by a small set of base patches. If the patches are writtenas vectors, then the collection of patch-vectors are (highly) linearly dependent, andthus have low rank(see Fig. 1). In this paper, for the pre-divided small patches withsize p1 × p2, we adopt the patch-rank as the sparsity regularization.

φ(x) = |Patch(x)|∗ (2)

where | · |∗ is the nuclear norm, which approximately represents the rank of amatrix.

Since we are fitting Poisson intensities, which are positive by nature. So we needadd a positive constraints ιC, which is the indicator function of the closed convex setC. In this case, C is the positive orthant.

Putting together the data fidelity term (1), the sparsity prior term (2), and thepositive constraints ιC, the proposed denoising problem is formulated as following

xi = minx

−λ1 ln L(xi) + λ2|Patch(xi)|∗ + ιC (3)

It’s easy to obtain the existence and uniqueness of the proposed method.


Proposition 1 Existence and Uniqueness

– Existence: As the functional (3) is coercive,and lower semi-continuous, the mini-mum solution M exists.

– Uniqueness: As the functional is convex, and the data-f idelity term − ln L(x) isstrictly convex, So (3) has unique solution.

4 Numerical method

Proximal splitting [9] provides an efficient optimization algorithm for such problemas (3) which is composed by several convex functionals. The key idea is that theproximity operator of each individual function has some (relatively) convenientstructure, for example, the closed form, then the sum problem will be solved by thecombination of each individual function.

We reformulate the problem (3) as following

xi = min λ1 f1(xi) + λ2 f2(xi) + f3(xi)

where f1(xi) = − ln L(xi) is the data-fidelity term, f2(xi) is the low patch-rankregularization, and f3(xi) is the positive constraint.

The proximity operator of f1(x) has been proposed in [3],

prox f1 x =(

xi − 1 + √(xi − 1)2 + 4yi

2

)

1≤i≤n

(4)

In Hayden Schaffer and Stanley Osher [17], the closed-form of proximity operatorof f2(xi) is singular value thresholding,

xi = SVT(Px(n+1), λ2

)(5)

The proximity operator of f3 is a simple projector onto the positive orthant.We present experiments on natural images which have periodical textures and

contaminated by poisson noise. All the experiments are conducted and timed on thesame PC with an Intel Core i5 2.50 GHz CPU that has 2 cores and 2 GB memory,running Windows 7 and Matlab (Version 7.10).

In this experiment, we choose two test images as shown in Fig. 1, one buildingimage and one fabric image, which both have periodical textures. Firstly the testimages should be resized to 256 × 256, and then the images contaminated by poissonnoise are simulated by Matlab, finally the performance of our proposed algorithmwill be compared with two other methods: total variation regularization, the point-wise method [6] and the Nonlocal-based (BM3D) [4, 14]. In our experiments, theparameters are set as λ1 = 0.01;,λ2 = 0.5λ1, and for the ‘building’ image, the patchsize is set as 64, for the ‘fabric’ image, the patch size is set as 12. We make some moreexperiments to compare the correlation of patch size and the final results, as shownin Fig. 3.

From the experiments, we found that for images with size 256 × 256, the pro-posed algorithm converge within at most 20 iterations (Fig. 4), and cost about


image with poisson noise patch size: 12 patch size: 32 patch size: 64

PSNR = 46.6975PSNR = 40.2125PSNR = 37.2310

PSNR = 39.7578 PSNR = 30.1123 PSNR = 21.5051

Fig. 3 Results of experiments with different patch size

3∼5 s. The results of the proposed algorithm and the comparisons are shown inFig. 5 and 6.

We use the standard Peak Signal to Noise (PSNR) to quantify the performance ofdenoising

PSNR(u, v) = 10 log10

(1

||u − v||2)

The corresponding PSNR value are shown in Table 1.From the experiments we can find that the proposed method via patch estimation

and low-rank regularization achieve more ‘smooth’ and ‘natural’ image, while themethod in [6] using pointwise estimation suffer from the ‘blurring’ effects. And

Fig. 4 Convergence curve ofproposed algorithm

0 5 10 15 200

5000

10000

15000

iterations

itera

tive

erro

r


original image contaminated by poisson noise TV regularization

BM3D pointwise denoising proposed method

Fig. 5 Result of experiments for denoising. The f irst row is the original image, image contaminatedby poisson noise, the results by TV, the second row is the results by BM3D, pointwise estimation[18], and the proposed method

original image contaminated by poisson noise TV regularization

BM3D pointwise denoising proposed method

Fig. 6 Result of experiments for denoising. The f irst row is the original image, image contaminatedby poisson noise, the results by TV, the second row is the results by BM3D, pointwise estimation[18], and the proposed method

Table 1 PSNR of different algorithm

Algorithm TV BM3D Pointwise algorithm from [6] Proposed algorithm

PSNR(building image) 38.2460 45.2530 41.2475 46.6975PSNR(fabric image) 39.2235 41.1554 33.6990 39.7578


practically, the textures of images we chosen aren’t strictly periodical, and theexperiments can also achieve satisfied results.

5 Conclusions

In this paper, we focus on one special class of images which have periodical texturesand contaminated by poisson noise. The main contribution includes: We propose animage denoising algorithm using patch estimation and low patch-rank regularization,and analyzes the existence and uniqueness property of the algorithm, and finallydesign an efficient numerical algorithm based on the splitting algorithm. The patch-based poisson likelihood functional effectively remove the ‘blurring’ effect, and byadopting the low patch-rank as the regularization, avoids the choosing of dictionary.Experiments show that our method performs well for this special class of imageswhich have periodical texture, and it does not require strictly periodical textures.

References

1. Besbeas P, Fies ID, Sapatinas T (2004) Comparative simulation study of wavelet shrinkageetimators for Poisson counts. Int Stat Rev 72:209–237

2. Chandrasekharan V, Sanghavi S, Parillo P, Wilsky A (2011) Rank-sparsity incoherence formatrix decomposition. SIAM J Optim 21(2):572–596

3. Combettes PL, Pesquet J (2007) A Douglas-Rachford splittting approach to nonsmooth convexvariational signal recovery. IEEE J Sel Topics Signal Process 1(4):564–574

4. Dabov K, et al (2007) Image denoising by sparse 3-D transform-domain collaborative filtering.IEEE Trans Image Process 16(8):2080–2095

5. Donoho D (1993) Nonlinear wavelet methods for recovery of signals, densities and spec-tra from indirect and noisy data. In: Proceedings of symposia in applied mathematics:differentperspectives on wavelets, American Mathematical Society, pp 173–205

6. Dupe F-X, Fadili J, Starck J-L (2011) Deconvolution under Poisson noise using exact data fidelityand synthesis or analysis sparsity priors. arXiv:1103.2213v1

7. Elad M, Figueiredo MAT, Ma Y (2010) On the role of sparse and redundant representationsin image processing. IEEE Proc - Special Issue on Applications of Sparse Representation andCompressive Sensing 98(6):972–982

8. Yin W, Hale E, Zhang Y (2008) Fixed-point continuation for L1-minimization: methodology andconvergence. SIAM J Optim 19(3):1107–1130

9. Goldstein T, Osher S (2009) The split Bregman method for L1-regularized problems. SIAM JImaging Sci 2(2):323–343

10. Green M (2002) Statistics of images, the TV algorithm of Rudin-Osher-Fatemi for image denois-ing and an improved denoising algorithm. CAM Report 02–55, UCLA

11. Jonsson E, Huang C-S, Chan T (1998) Total variation regularization in positron emission tomog-raphy. CAM Report 98–48, UCLA

12. Kervrann C, Trubuil A (2004) An adaptive window approach for poisson noise reduction andstructure preserving in confocal microscopy. In: International Symposium on Biomedical Imag-ing (ISBI04), Arlington, VA

13. Kolaczyk E (1999) Wavelet shrinkage estimation of certain Poisson intensity signals using cor-rected thresholds. Statist Sinica 9:119–135

14. Markku M, Foi A (2011) Optimal inversion of the Anscombe transformation in low-countpoisson image denoising. IEEE Trans Image Process 20(1):99–109

15. Rudin L, Osher S, Fatemi E (1992) Nonlinear total variation based noise removal algorithms.Phys D 60:259–268

16. Rudin LI, Osher S (1994) Total variation based image restoration with free local constraints. In:ICIP (1), pp 31–35

17. Schaeffer H, Osher S (2011) A low patch-rank interpretation of texture.ftp://ftp.math.ucla.edu/pub/camreport/cam11-75.pdf

http://arxiv.org/abs/1103.2213v1

ftp://ftp.math.ucla.edu/pub/camreport/cam11-75.pdf


18. Timmermann K, Novak R (1999) Multiscale modeling and estimation of Poisson processes withapplications to photonlimited imaging. IEEE Trans Inform Theory 45:846–852

19. Wright J, Ganesh A, Rao S, Ma Y (2009) Robust principal component analysis: exact recoveryof corrupted low-rank matrices via convex optimization. In: Proceedings of the Conference onNeural Information Processing Systems (NIPS)

20. Wright J, Ganesh A, Rao S, Ma Y (2011) Robust principal component analysis? J ACM58(3),11:1–37

21. Wright J, Ma Y, Mairal J, Sapiro G, Huang T, Yan SC (2010) Sparse representation for computervision and pattern recognition. Proc IEEE 98(6):1031–1044

22. Yin W, Osher S, Goldfarb D, Darbon J (2008) Bregman iterative algorithms for L1-minimizationwith applications to compressed sensing. SIAM J Imaging Sci 1:143–168

Bo Li received the Ph.D. degree in Mathematics from Mathematics Department, Dalian Universityof Technology, Dalian, in 2008. He is the associate Professor at College of Mathematics andInformation Science, Nanchang Hangkong University in China. His research interests include inverseproblems in image processing and content-based image retrieval.

Ge Lin received the Ph.D. degree in computer application technology from Sun Yat-Sen Universityin 2011. He is a postdoctoral researcher at college of Information science and technology, SunYat-Sen University in China. His research interests include CAD, computer simulation and digitallife.

Qiang Chen received the B.S. degree in Mathematics from Mathematics Department, South ChinaNormal University, Guangzhou, in 1984. He is a Professor at Dept.of Computer Science, GuangdongUniversity of Education in China. His research interests include data base system, content-basedvisual information retrieval and vision perception., knowledge management.

Hongyi Wang received the B.S. degree in Mathematics from Mathematics Department, AnaingNormal University. He is now a master student of Nanchang Hangkong University. His researchinterests include image processing, computer graphics.

Documents

Image denoising with patch estimation and low patch-rank regularization