Image Deblurring Blind Deconvolutionbased on Sparsity Regularization

Lili ChenDepartment of Electrical

and Computer EngineeringCollege of EngineeringUniversity of MichiganAnn Arbor, Michigan

Email: chenlili@umich.edu

Mingjie GaoDepartment of Electrical

and Computer EngineeringCollege of EngineeringUniversity of MichiganAnn Arbor, Michigan

Email: gmingjie@umich.edu

Wenhao PengDepartment of Electrical

and Computer EngineeringCollege of EngineeringUniversity of MichiganAnn Arbor, Michigan

Email: pwhum@umich.edu

Abstract—Blind deconvolution in image deblurring is an open-ended problem because various heuristic assumptions can beposed on the point-spread function and the sharp image. In thisproject, we explore the effectiveness of a new heuristic assumptionthat the image is sparse in terms of its gradient. This heuristicfavors sharpness of the recovered image because the edges aresparse. The point-spread function is assumed to be non-negative.Using iterative-shrinkage-thresholding algorithm, the non-convexcost function for the estimated gradient of sharp images is solved.Using iterative-reweighted-least-square algorithm, the norm-2regression problem for point-spread functions is solved. Thealgorithm is implemented and then tested in this project. Byblurring each test image using various point spread functions,feeding the blurred images into our implementation, and thencomparing the estimated point-spread function with the actualpoint-spread function using SSIM, we find that the algorithmperforms pretty well on Gaussian noise and motion blur.

I. INTRODUCTION

A. Problem Statement

Images are prone to being blurred. For instance, takingsnapshots out-of-focus will lead to blurring. Sometimes, al-though all setups go well, images may still not be clearenough because of systematic aberration (spherical aberration,coma, astigmatism and etc.) of lens set in the camera. Withoutknowing the way the image is blurred (the blurring kernel isunknown), traditional Wiener filters cannot be applied. Underthese conditions, blind deconvolution shows its importance.However, it is quite challenging to recover images withoutknowing the original blurring kernel. To realize the blinddeconvolution function, some assumptions like sparsity of highfrequency components are needed.

B. Current Method-MATLAB deconvblind Function

The most common and simplest way to do blind deconvolu-tion to recover from the blurred figure is to use MATLAB built-in functions. In MATLAB Image Processing Toolbox,there is one useful function called deconvblind. In thisfunction, it takes the blurred image I and the initial point-spread function INITPSF as input:

[J PSF ] = deconvblind(I, INITPSF,NUMIT ). (1)

NUMIT is the number of iterations, J is the recovered imageand PSF is the restored point-spread function.

This algorithm deconvolves I using the maximum likelihoodmethod. Estimators are constrained by positivity and sum of allelements should be one. It also tries to estimate the point-spreadfunction and deconvolves using Lucy-Richardson algorithm.Lucy-Richardson algorithm deconvolves the blurred image witha known kernel. By setting cost function properly and tryingiteratively to minimize cost function, it can converge to onelocal extrema point [1].

C. Problems with MATLAB deconvblind Function

Actually, the result of MATLAB deconvblind is largelybased on the initial point-spread function people guess. Sincepeople often have little idea of the blurring method, thePSF they guess may be quite different from the real one. Ifpeople simply input a normalized uniform PSF into MATLABdeconvblind, the result can be even worse than the originalblurred image. Another problem with this MATLAB built-infunction is that it cannot recover image with motion blurwell. Deconvblind tends to ruin the figure if the inputINITPSF is not similar to the real one and here is anexample that exposes these problems. Figure 1 is our testimage and blurred image. To find the effect of motion blur tocharacters, the blurred figure is made by adding a 45◦ motionblur. Figure 2 shows the problem. After deconvolving withMATLAB deconvblind function using the original sharpfigure, characters on the original sharp can barely be figuredout. Also, the blurred figure leads to even worse result. It doesnot solve the problem of blurring but ruins all characters.

Another problem of MATLAB built-in deconvblind func-tion is that it often leads to a non-optimal result because ofsystematic error in that function. In that cost function, it triesto converge to one extrema point. However, for non-convex andnon-concave curves, local extrema may differ a lot from theglobal extrema [1]. Although increasing number of iterationsand trying different variations of estimators can reduce thiskind of error, this systematic error is still quite annoying. Itcosts more time and it cannot always give better results.

Fig. 1. The test image we use to showcase problems in MATLABdeconvblind. The lower one is the image blurred from the upper testimage with a 45◦ motion blur.

Fig. 2. Solution of MATLAB deconvblind function with input originalsharp image in Figure 1. It actually ruins the image. Characters can be barelyfigured out.

Fig. 3. Solution of MATLAB deconvblind function with input blurredimage in figure 1. It also ruins the image. Characters can be barely figuredout. It gives worse result compared to Figure 2 because it not only does notsolve the problem of blurring, but creates new problems.

II. METHODS

A. Programming Environment and Explanation

The programming environment of our project is MATLAB,release R2017a. The main advantage is that it is fast onmatrix operations, and it provides a lot of toolboxes. ImageProcessing Toolbox helps us read in and display imagesand provides a baseline reference function deconvblind.

Its fspecial is used for creating Gaussian and Motionblurring kernels. Its conv2 is used for performing convolutionoperations of images. SSIM is used for evaluating how similarthe estimated blurring kernel is compared to the actual blurringkernel.

B. Mathematical Formulas and Derivations

1) Blurring Model: The blurring model is described byEquation 2.

y = x⊗ k (2)

where y is the blurred image, x is the sharp image, and k isthe blurring kernel.

2) Converting the Model to Gradient: Notice that boththe convolution operator and the gradient operator are linearoperators. Therefore, we can apply a gradient to the imagesx and y first, and retain their convolution relation. The newrelation is described by Equation 3

∇y = (∇x)⊗ k (3)

3) Heuristic Assumptions: Notice that many blurring kernelsare non-negative. For instance, when a motion blur is invoked,the original image is essentially replicated along some direction,producing a non-negative kernel. Also notice that in a readableimage, the edges are sparse. In an image, edges are identifiedby a sharp contrast between adjacent pixels. Therefore, for ablind deconvolution, without any information on the kernel orthe sharp image, we make the following heuristic assumptions:• The blurring kernel is non-negative.• The gradient of the sharp image is sparse.4) Cost Function: The cost function [2] to be minimized is

given in Equation 4

f = λ||(∇x)⊗ k −∇y||22︸ ︷︷ ︸Difference

+||∇x||1||∇x||2︸ ︷︷ ︸

Sparsity Regularization

(4)

We minimize this cost function such that• Convolution of estimated gradient of the sharp image with

estimated blurring kernel is close to the gradient of theblurred image.

• Estimated gradient of the sharp image is sparse.5) Kernel Estimation: Our algorithm for a single scale kernel

estimation is given in Algorithm 1. While both updating ∇xand updating k are trying to lower the same cost function, theirmathematical models are fundamentally different. For updateon ∇x, this cost function is non-convex, yet for update on k,the problem is just an Lp norm linear regression.

For update on ∇x, our algorithm uses1) gradient descend to lower the difference term, and2) iterative shrinkage and thresholding algorithm (ISTA [3])

to lower the sparsity regularization term.If the overall effect of previous two steps does not actuallylower the cost function, it will try different ISTA steps.

To update k, iterative reweighted least square algorithm(IRLS [4]) is used.

Using this single scale implementation, the algorithm down-scales the blurred image and the estimated kernel to the lowestlevel first, and then interpolates the crude result to obtain theinitial condition for the next scale.

Algorithm 1 (∇x) and k updateInput: ∇y, (∇x)j , kj , λOutput: (∇x)j+1, kj+1

for itxk=1 to 21 docurrent cost=λ||(∇x)j ⊗ k −∇y||2 + ||(∇x)j ||1||(∇x)j ||2δ = 1× 10−3

while δ > 1× 10−4 do(∇x)j+1 = (∇x)jfor itx=1 to 2 doβ = λ||(∇x)j+1||2for itista=1 to 2 do

descend = βδ[(∇y) − ((∇x)j+1 ⊗ kj)] ⊗rot90(kj , 2)descended = (∇x)j+1+descendshrinked= (∇xj+1) + descend− δthresholded= max(|shrinked|, 0)sign(descended)

end forend fornext cost= λ||(∇x)j+1 ⊗ k −∇y||2 + ||(∇x)j+1||1

||(∇x)j ||2if next cost> 3×current cost thenδ = δ/2

elsebreak

end ifend whileAk = [(∇x)j+1 ⊗ kj ]⊗ (rot90((∇x)j+1, 2))r = rot90((∇x)j+1)⊗ (∇y)−Akρ = r′(:)r(:)Ap = [(∇x)j+1 ⊗ r]⊗ rot90((∇x)j+1, 2)α = ρ/(r(:)′Ap(:))kj+1 = kj + αrif ρ < 1× 10−4 then

breakend ifkj+1(kj+1 < 0) = 0kj+1 = kj+1./sum(sum(kj+1))

end for

6) Image Recovery: Simple deconvolution algorithms[2]like Richardson-Lucy are sensitive to a wrong kernel estimate.Therefore, we implement deconvolution algorithm by ourselves.

The cost function[5] we want to minimize is

f =λ

2||x⊗ k − x||2 + β

2||w −∇x− b||2 + ||w||2 (5)

where k is the kernel, x is the recovered image, b, w areauxiliary variables and λ, β are weights. At the beginning, x isassigned to be the blurred input image. Then we do iterationsto update w, b and x.

a) Update w: f is a quadratic function with respect tow. Hence the value of w which gives minimum f is

w = ∇x+ b− 1

β(6)

b) Update b: f is a linear function with respect to b.Hence the value of b which gives minimum f is

b = w −∇x (7)

c) Update x: We use Equation (4) in [5] to directlycalculate the optimal x.

7) Implementation: For practical purposes, the ∇ operatoris taken in the vertical and horizontal sense, and their averageacts as the gradient term in the cost function. After studying thedemonstration code [6] of the author, we implement the singlescale blind deconvoultion ss_blind_deconv_wp.m andnon-blind deconvolution nb_deconv_gmj.m on our own byunderstanding the contents of the reference paper, and adaptits multi-scale implementation to improve performance. Thetestsuite.m is created for running through our dataset [7]and calculating SSIM.

8) Obtaining Results: Our results are obtained by spec-ifying contents in testsuite.m. Blurring kernels aredefined in h, and file names are specified in picname.It is assumed that blurring kernel size is less than 31by 31, so that the initial kernel size is fixed to 31.This can be easily changed by modifying the call todeconv_blind_sparsity_regularization, specify-ing another kernel size (side length). The kernel size has to bean odd number to prevent alignment issues with convolution.The scripts will run through all pictures specified in picnameover all blurring kernels h, calculate SNR and SSIM, and storethe calculated results into binary files, and SNR, SSIM valuesinto text files. To summarize, we test our algorithm using thefollowing steps.

1) Use MATLAB to produce blurring kernels.2) Apply the blurring kernels to the original test images to

get the blurred images.3) Send the blurred images into our function.4) Get deblurred images and the estimated kernels returned.5) Evaluate estimated kernels with respect to original

blurring kernels using SSIM.

III. RESULTS

A. Blurring Kernels

We first produce 6 kernels in MATLAB.• Kernel h1 is a 31-by-31 Gaussian lowpass filter with

standard deviation 2 created by fspecial function.• Kernel h2 is a 5-by-5 averaging kernel.

h2 =1

25

1 1 1 1 11 1 1 1 11 1 1 1 11 1 1 1 11 1 1 1 1

(8)

• Kernel h3 is a 5-by-5 highpass filter.

h3 =1

8

−1 −1 −1 −1 −1−1 2 2 2 −1−1 2 8 2 −1−1 2 2 2 −1−1 −1 −1 −1 −1

(9)

Fig. 4. Visualization of the six kernels used for blurring test images. (a)Gaussian lowpass filter kernel. (b) Averaging kernel. (c) Weak 45◦ motionblur kernel. (d) Horizontal motion blur kernel. (e) 45◦ motion blur kernel.Note that kernel h3 is not drawn because it contains negative components.

Fig. 5. Our selected test image [7]. This image is blurred and then recoveredusing our algorithm.

• Kernel h4 is a weak version of 7-by-7 45◦ motion blur.

h4 =1

7

1 0 0 0 0 0 00 1 0 0 0 0 00 0 1 0 0 0 00 0 0 1 0 0 00 0 0 0 1 0 00 0 0 0 0 1 00 0 0 0 0 0 1

(10)

• Kernel h5 is a horizontal motion blur with 20-pixel motioncreated by fspecial function.

• Kernel h6 is a 45◦ motion blur with 20-pixel motioncreated by fspecial function.

The six kernels are visualized in Figure 4.

B. Blurring the Test Images

We select one test image from our dataset for demonstration.The original image is shown in Figure 5.

Now we put Figure 5 into blurring kernels h1 to h6 and theresults are shown in Figure 6.

C. Deblurring the Images

We send the blurred images into our function written inMATLAB. The function will deblur the images using ouralgorithm and return the deblurred images as well as the

Fig. 6. Blurred images for the test image. (a) Blurred image from h1. (b)Blurred image from h2. (c) Blurred image from h3. (d) Blurred image fromh4. (e) Blurred image from h5. (f) Blurred image from h6.

Fig. 7. Deblurred images using our algorithm on the blurred test images. (a)Deblurred image from h1. (b) Deblurred image from h2. (c) Deblurred imagefrom h3. (d) Deblurred image from h4. (e) Deblurred image from h5. (f)Deblurred image from h6.

estimated kernels. The deblurred images are shown in Figure 7and the estimated kernels are shown in Figure 8.

D. Structural Similarity Index (SSIM)

We use structural similarity index (SSIM) to evaluate thequality of our algorithm. SSIM is a weighted combination ofthree comparative measurements between the input image andthe reference image:

SSIM = lα · cβ · sγ ∈ [0, 1] (11)

where l is luminance, c is contrast, s is structure and α, β, γ areweights. The larger the value is, the more similar two imagesare. SSIM is more consistent with human visual perceptioncompared to traditional measures such as signal-to-noise-ratioor mean square error [8].

The SSIM values of our estimated kernels with respect tothe original kernels are shown in Table I.

Fig. 8. Estimated kernels returned by our algorithm when inputs are blurredversions of Figure 5. (a) Estimated kernel h′1 for h1. (b) Estimated kernel h′2for h2. (c) Estimated kernel h′3 for h3. (d) Estimated kernel h′4 for h4. (e)Estimated kernel h′5 for h5. (f) Estimated kernel h′6 for h6.

TABLE IESTIMATED KERNEL SSIM

Estimated kernel SSIM

h′1 0.995

h′2 0.101

h′3 0.006

h′4 0.608

h′5 0.901

h′6 0.857

E. Explanation of the Results

1) Gaussian Blur Kernel h1: Compare Figure 8(a) withFigure 4(a). Our estimated kernel is very similar to the originalkernel. SSIM value is 0.995, which is very close to 1. Thedeblurred image Figure 7(a) is almost the same as the originalimage. It means that our algorithm is good at solving Gaussianblurring kernel.

2) Averaging Kernel h2: Compare Figure 8(b) with Fig-ure 4(b). The original kernel is an averaging kernel, but ourestimation is more like a Gaussian kernel. SSIM value is0.101, which is not very good. However, the deblurred imageFigure 7(b) looks very similar to the original image. In fact,both Gaussian kernel and averaging kernel make the imagesmoother. We can regard Gaussian kernel as zero-extendedaveraging kernel. Hence it is not a surprise that our algorithmreturns an estimation similar to a Gaussian kernel for theaveraging kernel in Figure 4(b).

3) High-frequency Blur Kernel h3: This case shows alimitation of our algorithm. As is mentioned before, we have theassumption that the blurring kernel is non-negative. However,there are kernels which contain negative components, such asthis high-frequency kernel h3. Our algorithm fails to estimatethe kernel and returns an estimation which is totally differentfrom the original kernel. SSIM value is only 0.006.

As for the images, compare Figure 5, Figure 6(c) andFigure 7(c). The high-frequency kernel makes the originalimage sharper. Our estimation returns a kernel like Gaussiankernel, thus making the blurred image even sharper.

4) Weak 45◦ Motion Blur Kernel h4: Compare Figure 8(d)with Figure 4(c). The estimated kernel shows the overall trendof the blurring kernel, but it is much shorter and more unclear.This is for lack of number of iterations. SSIM value is givenas 0.608. The quality of deblurred image Figure 7(d) is muchbetter than that of the blurred image Figure 6(d).

5) Horizontal Motion Blur Kernel h5: Compare Figure 8(e)with Figure 4(d). The estimated kernel shows the overall trendof the blurring kernel, but like in Section III-E4, it is shorterand more unclear. This is also for lack of iteration numbers.SSIM value is 0.901, which is close to 1, meaning that ourestimated kernel resembles the original kernel. The deblurredimage Figure 7(e) is clearer than blurred image Figure 6(e).

6) 45◦ Motion Blur Kernel h6: Compare Figure 8(f) withFigure 4(e). We can recognize a diagonal line in the estimatedkernel, but still, it is not long and clear enough. SSIM value is0.857. The deblurred image Figure 7(f) is clearer than blurredimage Figure 6(f).

Note that in Figure 7(e) and Figure 7(f), there are severewave noises. The reason is that blurring kernels h5 and h6 aregenerated using fspecial with 20-pixel motion. However,the pixel motion is not even. Contrary to weak motion blurkernel h4, h5 and h6 also have fringing effect. Hence, if thealgorithm fails to estimate the edges of the kernels correctly,recovering the images using these kernels will lead to wrongpixel shift amount and the effect will be shown as wave noisesin the figures.

F. Other Test Images SSIM

We repeat test steps on all our twelve test images and weget a set of SSIM values. The values are shown in Table II.The values are quite similar to those in Table I and can beexplained in the same way.

For reference, we calculate SSIM values for estimated kernelsfrom deconvblind with respect to original kernels. The

TABLE IISSIM VALUE OF ESTIMATED KERNELS FROM OUR ALGORITHM

SSIM value

Test image No. h′1 h′2 h′3 h′4 h′5 h′61 0.995 0.101 0.006 0.608 0.901 0.857

2 0.996 0.097 0.006 0.613 0.918 0.858

3 0.997 0.101 0.006 0.614 0.923 0.850

4 0.997 0.096 0.007 0.614 0.931 0.849

5 0.996 0.098 0.006 0.611 0.917 0.860

6 0.996 0.100 0.006 0.609 0.925 0.856

7 0.998 0.097 0.006 0.611 0.926 0.865

8 0.993 0.102 0.007 0.609 0.912 0.854

9 0.999 0.097 0.007 0.612 0.911 0.860

10 0.993 0.100 0.005 0.610 0.900 0.857

11 0.995 0.102 0.006 0.609 0.905 0.855

12 0.991 0.104 0.006 0.608 0.912 0.842

TABLE IIISSIM VALUE OF ESTIMATED KERNELS FROM DECONVBLIND

SSIM value

Test image No. h′1 h′2 h′3 h′4 h′5 h′61 0.957 0.108 0.003 0.593 0.884 0.821

2 0.958 0.108 0.004 0.593 0.885 0.820

3 0.957 0.108 0.003 0.593 0.886 0.818

4 0.956 0.108 0.003 0.593 0.885 0.818

5 0.960 0.108 0.004 0.595 0.888 0.820

6 0.957 0.108 0.003 0.593 0.885 0.818

7 0.956 0.108 0.003 0.593 0.885 0.818

8 0.958 0.108 0.003 0.593 0.885 0.820

9 0.958 0.108 0.004 0.593 0.885 0.820

10 0.958 0.108 0.004 0.594 0.888 0.819

11 0.957 0.108 0.003 0.593 0.884 0.819

12 0.956 0.108 0.003 0.593 0.884 0.818

results are in Table III. We can see except for h′2, SSIM valuesin Table II are larger than SSIM values in Table III. Overall,our algorithm returns better estimated kernels compared toMATLAB deconvblind function.

G. Solving the Problems in Section I-C

Go back to the problems mentioned in Section I-C. If we putimages in Figure 1 into our algorithm, we can get output imagesshown in Figure 9 and Figure 10. Our algorithm manages tokeep the image nearly unchanged if input is already clear anddeblur the image if input is blurred.

IV. CONCLUSION

A. Conclusion and Discussion from Our Results

Our project focuses on blind deconvolution algorithm.Iterative shrinkage and thresholding algorithm (ISTA) anditerative reweighted least square algorithm (IRLS) are appliedfor updating our estimated gradient of our blurred image andupdating our estimated blurring kernel. From results shown inthe previous section, we can see that our algorithm is really

Fig. 9. Output of our algorithm with original sharp image in Figure 1 as input.Unlike deconvbind, our algorithm keeps the image nearly unchanged.

Fig. 10. Output of our algorithm with blurred image in Figure 1 as input. Itrecovers the image to some extent, making characters more recognizable.

good at recovering low-pass blurring kernel like Gaussian blurand motion blur. However, it still has some limitations.

First, based on our assumption, high frequency componentsof blurred image should be sparse. It is always the case formotion blur. However, for other kinds of noise like peppernoise, we cannot obtain good results from our algorithm sincepepper noise has too many high frequency components. If thereis a figure with both motion blur and pepper noise, one possiblemethod for image recovering is that we first use median filterfor pepper noise removal and than use our algorithm to removemotion blur.

Second, based on our algorithm for normalizing our esti-mated kernel, positivity is assumed. However, not every blurringkernel has only non-negative components. Actually, havingnegative components is quite common for high pass blurringkernels. For instance, the high pass kernel we use in our project(Kernel 3) is the case. Under this condition, output image afterdeblurring using our algorithm becomes sharper than it shouldbe. Thus, our result is undesirable for that case.

B. Benefits and Potential Application of Our Blind Deconvolu-tion Function

Indeed, blind deconvolution has very wide applications.And it can bring many benefits for people from amateurs toprofessionals. For example, in field of photography, shootingscenic photos out-of-focus can always lead to blurring andblurring is quite annoying. Blind deconvolution like ouralgorithm can help solve this problem. Also, in field ofmicroscopy, blurring due to poor resolution of microscope lensis very common. Instead of using complicated and expensivelens set to solve this problem, blind deconvolution like our

algorithm provides a naive but nice way for solving thisproblem.

ACKNOWLEDGMENT

The authors would like to thank Prof. Anastasopoulos andGSI Sawicki for providing instructions and help on this project.

REFERENCES

[1] F. Chen and J. Ma, “An Empirical Identification Method of Gaussian BlurParameter for Image Deblurring,” IEEE Trans. Signal Processing., vol.57, no. 7. pp. 2467-2478, Jul. 2009.

[2] D. Krishnan, T. Tay and R. Fergus, “Blind deconvolution using anormalized sparsity measure,” CVPR 2011, Providence, RI, 2011, pp.233-240.

[3] A. Beck and M. Teboulle. “A fast iterative shrinkage-thresholdingalgorithm for linear inverse problems,” SIAM J. on Img. Sciences, vol. 2,no. 1, pp. 183-202, Mar. 2009.

[4] A. Levin, R. Fergus, F. Durand, and W. Freeman. “Image and depth froma conventional camera with a coded aperture,” SIGGRAPH, vol. 26, no.3, pp. 70, Jul. 2007.

[5] D. Krishnan and R. Fergus. Fast image deconvolution using hyper-laplacianpriors. In NIPS, 2009.

[6] "Blind Deconvolution << Dilip Krishnan", Cs.nyu.edu, 2017. [On-line]. Available: http://cs.nyu.edu/~dilip/research/blind-deconvolution/.[Accessed: 21- Apr- 2017].

[7] "Image Databases", Imageprocessingplace.com, 2017. [Online]. Available:http://www.imageprocessingplace.com/root_files_V3/image_databases.htm. [Accessed: 21- Apr- 2017].

[8] Zhou Wang, A. C. Bovik, H. R. Sheikh and E. P. Simoncelli, “Imagequality assessment: from error visibility to structural similarity,” IEEETrans. Image Processing, vol. 13, no. 4, pp. 600-612, Apr. 2004.