study Coded Aperture

Embed Size (px)

DESCRIPTION

study

Text of study Coded Aperture

  • 1. study
    Image and Deoth from a Conventional Camera with a Coded Apertrue
    Anat Levin, Rob Fergus,Frdo Durand, William Freeman
    MIT CSAIL

2. Single input image
real objects
Coded Aperture
output #1: Depth map
output #2: all-infocusedimage
3. Conventional aperture and depth of field
Big aperture
Object
Focal plane
Small aperture
4. Depth from defocus
Camera sensor
Lens
Point spread function
Focal plane
http://groups.csail.mit.edu/graphics/CodedAperture/CodedAperture-LevinEtAl-SIGGRAPH07.ppt
5. Depth from defocus
Camera sensor
Lens
Object
Point spread function
Focal plane
http://groups.csail.mit.edu/graphics/CodedAperture/CodedAperture-LevinEtAl-SIGGRAPH07.ppt
6. Depth from defocus
Camera sensor
Lens
Object
Point spread function
Focal plane
http://groups.csail.mit.edu/graphics/CodedAperture/CodedAperture-LevinEtAl-SIGGRAPH07.ppt
7. Defocus as local convolution
Calibratedblur kernels at depth K
Local observed
sub-window
Sharp
sub-window
Input defocused image
Depth k=1
Depth k=2
Depth k=3
8. Introduction
Estimation of depth a branch of Computational Photography
Most challenges ofy = fk * x

  • Hard to de-convolve evenwhen kernel is known

Input
Ringing with the traditional Richardson-Lucyalgorithm

  • Hard to identify correct scale:

?
Larger scale
?
Correct scale
?
Smaller scale
9. Related work depth estimation
Active methods additional illumination sources

  • Structured light methods

Nayar et al. ICCV 95
Zhang and Nayar, SIGGRAPH 06
Projection Defocus Analysis for Capture and Image Display, Zhang and Nayar, 06
10. Related work depth estimation
Passive methods changes of focus

  • Depth from defocus (DFD)

Pentland, IEEE 87
Chaudhuri, Favaro et al. , 99

  • Blind Deconvolution image prior, maximum likelyhood

Kundur and Hatzinakos , IEEE 96
Levin,NIPS 06

  • Coded apertures for light gathering

Fenimore and Cannon, Optics 78
11. Related work depth estimation

  • Passive methods changes of viewpoints

12. Plenoptic /light field cameraAdelson and Wang, IEEE 92
Ng et al., 05

  • Wavefront coding

Cathey & Dowski, Optics 94, 95
1.Rays don't converge anymore
2.Image blur is the same for all depth
3.Blur spectrum does not have too many zeros
CompPhoto06/html/lecturenotes/25_LightField_6.pdf
13. Overview
Try deconvolving local input windows with different scaled filters:
?
Larger scale
?
Correct scale
?
Smaller scale
Somehow: select best scale
14. Challenges & contributions
Hard to de-convolve even when kernel is known
IDEA 1: Natural images prior
Hard to identify correct scale
IDEA 2: Coded Aperture
15. Deconvolution is ill posed
Solution 1:
=
?
Solution 2:
=
?
16. IDEA 1: Natural images prior
What makes images special?
Natural
Unnatural
Image
gradient
Natural images have sparse gradients
put a penalty on gradients
17. Deconvolution with prior
Convolution error
Derivatives prior
2
?
Low
Equal convolution error
2
?
High
18. Comparing deconvolution algorithms
Richardson-Lucy
Input
spread gradients
localizes gradients
Gaussian prior
Sparse prior
19. Statistical Model of Images
Deconvolution using natural image priors, Levin et. al., ETAI 07
Spatial domain
Frequency domain
20. Maximum a-posteriori P(x|y)
likelyhood
Image prior
(gradient here)
Gradient operator
For Gaussian priors
For sparse priors
21. Minimize deconvolution error
22. Deconvolution using a Gaussian prior
Note: solved in the frequency domain in a few seconds for MB size file
23. Deconvolution using a sparse prior
Using an iterative reweighted least squares process (IRLS) [Meer 2004; Levin and Weiss to appear]
Cannot solve in frequency domain
Note: solved in the frequency domainaround1 hour on 2.4Ghz CPR for 2MB file
24. Iterative reweighted least squares process (IRLS)
25. Recall: Overview
Try deconvolving local input windows with different scaled filters:
?
Larger scale
?
Correct scale
?
Smaller scale
Somehow: select best scale
Challenge: smaller scale not so different than correct
26. IDEA 2: Coded Aperture
Mask (code) in aperture plane
Make defocus patterns different from natural images and easier to discriminate
Conventional aperture
Our coded aperture
27. Lens with coded aperture
Image of a defocused point light source
Aperture pattern
Camera sensor
Lens with coded aperture
Object
Point spread function
Focal plane
28. Why coded ?
Coded aperture- reduce uncertainty in scale identification
Conventional
Coded
Larger scale
Correct scale
Smaller scale
29. Why coded ?
Coded aperture- reduce uncertainty in scale identification
Conventional
Coded
Larger scale
Correct scale
Smaller scale
30. Fourier transforms of 1D slide through the blur pattern
31. Coded aperture: Scale estimation and division by zero
spectrum
spectrum
spectrum
spectrum
spectrum
Frequency
Frequency
Frequency
Frequency
Frequency
Estimated image
?
Observed image
=
Filter, correct scale
Division by zero
Estimated image
?
spatial ringing
=
Filter, wrong scale
32. Division by zero with a conventional aperture ?
spectrum
spectrum
spectrum
spectrum
spectrum
Frequency
Frequency
Frequency
Frequency
Frequency
Estimated image
?
No zero at !
Observed image
=
Filter, correct scale
No zero at !
Tiny value at
no spatial ringing
Estimated image
?
=
Filter, wrong scale
is zero !
33. Filter Selection Criterion
The filter f has good depth discrimination - blurry image distributions Pk1(y) and Pk2(y) at depths k1 and k2 shouldnot be similar
KL-divergence scores
34. Filter Design
Practical constrains
Binary filter to construct accurately
Cut the filter from a single piece
Avoid excessive radial distortion
Avoid using the full aperture
Diffraction impose a min size on the holes in the file
Spec.
13x13 patterns with 1 mm holes
Each pattern, 8 differentscales
Varying between 5~15 pixels in width
35. Filter Design
Conventional
Conventional
36. Blur scale identification
Not robust at high-frequency noise
Un-normalized energy term
klearn to minimizethe scale misclassification error on a set of traning images
Ek is approximate by the reconstruction error by ML solution
x* is the deblurred image
37. Regularizing depth estimation
38. Results
39. Applications
Digital refocusing from a single image
e.g.Synthesis an all-focus image
e.g.Post-exposure
40. Conclusion
Pros.
All-infocus image and depth at a single shot
No loss of image resolution (compared with Plenoptic camera)
Simple modification
Coded aperture
Conventional aperture
Cons.
50 % light is blocked
Depth is coarse
May need manual correction