Sparsity-based Image Sparsity-based Image Deblurring with Deblurring with Locally Locally Adaptive Adaptive and and Nonlocally Nonlocally RobustRobust Regularization RegularizationWeisheng DongWeisheng Dongaa, Xin Li, Xin Libb, Lei , Lei

ZhangZhangcc, Guangming Shi, Guangming Shiaa

aaXidian University, Xidian University, bbWest Virginia West Virginia University, University, ccHongKong Polytechnic HongKong Polytechnic

UniversityUniversityThis work is partially supported by NSF CCF-0914353, HK RGC General Research Fund (PolyU 5375/09E), NSFC (No. 60736043,61072104, 61070138,and 61071170), and the Fundamental Research Funds of the Central Universities of China (No. K50510020003)

History of Image History of Image RestorationRestoration

(Non-blind Image (Non-blind Image Deconvolution)Deconvolution)Method ISNR(dB)

Forward: Fourier-wavelet regularized deconvolution for ill-conditioned systems (Neelamani, Choi, Baraniuk, TSP’2004)

7.30

An Expectation-Maxization algorithm for wavelet-based image restoration (Figueiredo,Nowak TIP’2003)

7.59

Total-Variation based image deconvolution: a majorization-minimization approach (Bioucas-Dias,Figueiredo,Oliveira ICASSP’2006)

8.52

A new TwIST: Two-step iterative shrinkage/thresholding algorithms for image restoration (Bioucas-Dias,Figueiredo TIP’2007)

8.63

Pointwise Shape-Adaptive DCT (Foi,Katkovnik,Egiazarian TIP’2007) 8.57

Block-matching and collaborative 3D filtering (Dabov,Foi,Katkovnik,Egiazarian TIP’2007) 8.34

Image restoration through L0 analysis-based sparse optimization in tight frames (Portilla ICIP’2009)

9.10

Variational Bayesian Image Restoration With a Product of Spatially Weighted Total Variation Image Priors (Chantas, Galatsanos, Molina, Katsaggelos, TIP 2010)

9.61

This work (sparsity-based image deblurring with local adaptive and nonlocal robust regularization)

9.00

Patch-based Image Deconvolution Via Joint Modeling of Sparse Priors (Jia, Evans ICIP’2011) 8.98

Fine-granularity and spatially-adaptive regularization for projection-based image deblurring (Li TIP’2011)

10.10

Centralized sparse representation for image restoration (Dong, Zhang, Shi ICCV’2011) 10.40Cameraman 256×256, 9×9 uniform blur, BSNR=40dB

Lessons We Have Lessons We Have LearnedLearned

““All models are wrong; but some are useful” – G. All models are wrong; but some are useful” – G. BoxBox Local models: wavelet/DCT, total-variation (TV), Local models: wavelet/DCT, total-variation (TV),

spatially-weighted TV (SWTV), …spatially-weighted TV (SWTV), … Nonlocal models: nonlocal-mean, BM3D, nonlocal Nonlocal models: nonlocal-mean, BM3D, nonlocal

TV, ASDS-AR-NL (precursor of this work), …TV, ASDS-AR-NL (precursor of this work), …

DCT>BM3D BM3D>DCT

One Simple MessageOne Simple Message

Local variation and nonlocal invariance Local variation and nonlocal invariance are two sides of the same coinare two sides of the same coin

Local variation

Nonlocal invariance

Kanizsa Triangle

Local View: Dictionary Local View: Dictionary LearningLearning

Daubechies’ wavelet,1988 Do&Vetterli’s contourlet,2005 Bell&Sejnowski’ICA,1996 Elad&Aharon’K-SVD,2006

2

2||||..

lts Φαx

0||||min lα

α

02||||||||)(min 2

llJ αΦαxα

Lagrange’s idea

12||||||||)(min 2

llJ αΦαxα

the magic of l1

NP-hard

HOTTY

NonlocalNonlocal View: Structural View: Structural ClusteringClustering

K

k Ciki

k

lJ

1

2

2||||)(min μxμ

Kmeans-based clustering

NLM denoising (Buades et al. CVPR’2005)

Variational FormulationVariational Formulation

K

k Cilki

N

i

m

jlij

k

lJ

1

1 1

2

1

12

||||

||||||||),(min

μΦα

αHDαxμα

2

1 1

2

2

12

||)(||

||||||||)(min

l

l

N

i

m

jlijJ

DαWI

αHDαxα

02||||||||)(min 2

llJ αHDαxα

NL-similarity penalty term Structural clustering penalty term

Key DerivationsKey Derivations

2

1 1

2

212||)(||||||||||)(min

ll

N

i

m

jlijJ DαWIαHDαxα

N

i

m

jlijl

J1 1

2

12||||||~||)(min αKDαyα

)()(,

0~

BI

H

WI

HK

yy

Iterative thresholding (via surrogate functions)

Typo in the paper

Summary of AlgorithmSummary of Algorithm

Connection with Other Connection with Other Competing WorksCompeting Works

Patch-based Image Deconvolution Via Joint Patch-based Image Deconvolution Via Joint Modeling of Sparse Priors (ICIP’2011)Modeling of Sparse Priors (ICIP’2011)

Nonlocal total-variation for image Nonlocal total-variation for image restoration (UCLA Math TR)restoration (UCLA Math TR)

Deconvolution network (CVPR’2010)Deconvolution network (CVPR’2010) Handling Outliers in Non-Blind Image Handling Outliers in Non-Blind Image

Deconvolution (ICCV”2011)Deconvolution (ICCV”2011) Close the Loop: Joint Blind Image Close the Loop: Joint Blind Image

Restoration and Recognition with Sparse Restoration and Recognition with Sparse Representation Prior (ICCV’2011)Representation Prior (ICCV’2011)

Experimental ResultsExperimental Results

MATLAB codes accompanying this work are availableFrom my homepage: http://www.csee.wvu.edu/~xinl/

Image Comparison Image Comparison Results (I)Results (I)

original

Noisy and blurred

SWTV (28.96dB)

L0-sparsity (29.04dB)

BM3D (30.22dB)

LANL (31.33dB)

Image Comparison Image Comparison Results (II)Results (II)

original SWTV (27.96dB) BM3D (27.22dB)

Noisy and blurred L0-sparsity (27.12dB) LANL (29.15dB)

Image Comparison Image Comparison Results (III)Results (III)

2

2||)(||

lDαWI

K

k Cilki

k11|||| μΦα

LANL (31.33dB) CSR (32.09dB)

LANL (29.15dB) CSR (29.75dB)

Conclusions and Conclusions and PerspectivesPerspectives

What should we care about?What should we care about? Pursue even higher ISNR value for cameraman?Pursue even higher ISNR value for cameraman? A collection of benchmark images? A collection of benchmark images? Landweber vs. Lucy-RichardsonLandweber vs. Lucy-Richardson

Application side: Application side: Motion deblurring: from non-blind to blind Motion deblurring: from non-blind to blind

image deconvolution?image deconvolution? What will be the next episode like the What will be the next episode like the

malfunctioned mirror of Hubble Space malfunctioned mirror of Hubble Space Telescope?Telescope?

Dehazing: from linear blur to nonlinear hazingDehazing: from linear blur to nonlinear hazing