Shaojie Zhuo, Dong Guo, Terence Sim

School of Computing, National University of Singapore

CVPR2010

Robust Flash Deblurring

Reporter: 周 澄 (A.J.)01/16/2011

Key words: image deblur, flash/no-flash technique, blur kernel estimate, sharp image reconstruction

•Goal•Major contributions•Related work•Flash deblurring framework‐MAP optimization•Practical implementation•Results•Limitations•Future works

Outline

•Deblur a shaken image by using a corresponding flash image generated from a conventional hand-held camera.

Goal

•Propose a novel approach by adding a flash image as a key constraint.•Well handle both the flash artifacts and the deconvolution artifacts.‐ Additional constraints introduced.•High quality results.‐ Insensitive to noise, fine image details.

Major contributions

•Non-blind image deconvolution (Single image method).Def:

Given the estimated blur kernel, the second step is to reconstruct a sharp image from the blurred image.

Related work

•[Fergus et al. TOG2006] Removing camera shake from a single photograph.Used a variational Bayes inference method with natural image statistics to estimate the motion blur kernel.

Related work

•[Jia. CVPR 2007] Single image motion deblurring using transparency.Investigated the relationship between object boundary transparency and the image motion blur and estimated the blur kernel from the alpha matte of motion objects.

Related work

•[Shan et al. TOG 2008] High-quality motion deblurring from a single image.Formulated the deblurring problem as an MAP problem and proposed a high-order derivatives image noise model and a local image prior to avoid trivial solution.

Related work

•Fergus, Jia, and Shan’s method are all able to obtain accurate kernels when the blur is small.

=> Fail at large shaking & noise.

Related work

•Wiener filter and the Richardson-Lucy (RL) deconvolution.[Levin et al. CVPR2009] Understanding and evaluating blind deconvolution algorithms.

Related work

•Wiener filter and the Richardson-Lucy (RL) deconvolution.

=> Suffer from deconvolution artifacts such as amplified noise and ringing artifacts.

Related work

•Regularization methods.‐In order to reduce artifacts.[Wang et al. SIIMS2008] A new alternating minimization algorithm for total variation image reconstruction.

[Yuan et al. TOG2008] Progressive interscale and intra-scale non-blind image deconvolution.

Related work

•Regularization methods.

=> Lost fine image details since you cannot separate image details from artifacts like noise or ringing artifacts properly.

Related work

•Additional hardware(Hybrid camera).High resolution camera + low resolution video camera.

=> Some image details still lost due to non-invertible motion blur.

Related work

•Multiple images solution.[Yuan et al. TOG2007] Image deblurring with blurred/noisy image pairs.

[Chen et al. CVPR2008] Robust dual motion deblurring.

[YW Tai et al. CVPR2005] Local color transfer via probabilistic segmentation by expectation-maximization.

Related work

•Flash/no-flash technique.[Agrawal et al. TOG2005] Removing photography artifacts using gradient projection and flashexposure sampling.

[Petschnigg et al. TOG2004] Digital photography with flash and no-flash image pairs.

[Eisemann et al. TOG2004] Flash photography enhancement via intrinsic relighting.

Related work

•Traditional flash/no-flash technique.

=> Need good alignment between two images.

Related work

•Input:A blur image B and corresponding flash image F.

•Output:A visually pleasant sharp image I with least flash artifacts or deconvolution artifacts.

Flash deblurring framework

•Problem formulation:Given the blurred image B and flash image F, our goal is to estimate a blur kernel K and a sharp image I, so that I,K and B can be represented by the convolution model and the gradients of I are close to those in F.(We talk about gradients later)

Flash deblurring framework

Flash deblurring framework

Kernel estimation

Sharp image

reconstruction

Maximum-a-posteriori (MAP) framework

•B = I * K + n, where n is the image noise which modeled as a set of independent and identically distributed(i.i.d.) Gaussian noise.

The convolution model

•Problem formulation:

where L(． ) = - log (p(． ))

MAP optimization

•Likelihood term(SSD):

•Analyze the SSD of the estimated kernel convolution result and the blur image.

MAP optimization

•Kernel prior term:

where the parameter α≦1;α=0.8 used in the paper.

MAP optimization

•Key idea: The robust flash gradient constraint encourages the gradients of reconstructed image to be close to those in F, while at the locations of flash artifacts, ambient shadows or noise it allow their gradient to differ to avoid flash artifacts and keep ambient illumination.

Flash gradient constraint

•Flash gradient constraint: observation.

MAP optimization

1D scanlines of intensities and gradients in R channel of the three images.

In the gradient plot, ΔI(cyan) is much close to ΔF(magenta), which acts as a guide to reconstruct the sharp image I.

•Flash gradient constraint:

where is the Lorentzian robust estimator and ε is a predefined constant.

MAP optimization

•Objective function:

Here we have the likelihood term, flash gradient constraint and kernel prior term, where λf and λk are the used to balance the three terms.

MAP optimization

Flash deblurring framework

Kernel estimation

Sharp image

reconstruction

Maximum-a-posteriori (MAP) framework

Kernel estimation

Sharp image

reconstruction

•Fix K, we can estimate I by solving:

Where the weight of re-weight least square for each pixel i at each iteration:

Kernel estimation

Sharp image

reconstruction

•Fix I, we can estimate K by solving:

•Both equation can be solved by iterative re-weight least squares(IRLS).

Kernel estimation

Sharp image

reconstruction

•The two steps are alternated until diff(K) is smaller than the threshold.•To avoid local minimum when blur kernel is large, the kernel estimation is performed in a coarse-to-fine manner in the scale space.

Kernel estimation

Sharp image

reconstruction

•Three common artifacts.1. Flash artifact regions.2. λf is set to be large to

suppress noise or ringing artifacts.

3. Over-saturated regions.

Kernel estimation

Sharp image

reconstruction

•Build a mask image M to change the weight of flash gradient constraint locally.

Kernel estimation

Sharp image

reconstruction

•Flash artifacts detect.‐|| K * I - F ||2.‐Only need to manually mark the the flash shadow edge, since the shadow regions still contains useful gradients.

Kernel estimation

Sharp image

reconstruction

•Add in the sparse gradient constraint.The objective func becomes:

where “ο” denotes the pixel-wise multiplication operator.•Solved by IRLS.

•Take flash image first, and then use high speed capturing mode capture blur image.

•As the time between two shots is small, the the motion during the shots is basically a translation, which just causes a shift in the estimated blur kernel. Therefore, no image alignment is required.

Practical implementation

Resaults – kernel RMS error

Resaults

Resaults

Results

•It cannot handle the spatially invariant motion blur model.‐ Additional alignment needed.‐ Ex: Busy traffic.

•The exposure time between flash/no-flash images should be close.‐ Temporal incoherence.•Two images should share same aperture value.‐ May generate blur artifacts caused by different focus.

Limitations

•Extend to video.‐ Coherent preserve.•Support hybrid system such as combining with IR depth sensor or stereo sensor to get more information from image.

Future works