Portable Light Field Imaging: Extended Depth of Field - webdav

Joint Research Institute in Image and Signal ProcessingEdinburgh Research Partnership in Engineering and Mathematics

Sparse Signal Modelling and Compressed Sensing

TH

E

U N I V E R SI T

Y

OF

ED I N B U

RG

H

T. BlumensathInstitute for Digital Communications

Joint Research Institute for Signal and Image ProcessingThe University of Edinburgh

September, 2008

home · prev · next · page

17 December 2011 NIPS 2011 Machine Learning Meets Computational Photography

Portable Light Field Imaging: Extended Depth of Field, Aliasing

and Superresolution

Paolo Favaro

joint work with Tom BishopThis work has been supported by EPSRC grant EP/F023073/1(P)


Imaging sensors

2


Imaging sensors

2

•Traditional cameras are based on the design of the human eye


Imaging sensors

2


•Q: Is this optimal for all vision tasks?


Imaging sensors

2



•Other designs in nature:-simple eyes-pit eyes-pinholes-spherical lenses-multiple lenses-corneal refraction

-composite eyes-apposition-neural superposition-refracting superposition-reflecting superposition-parabolic superposition


Imaging sensors

2



•Other designs in nature:-simple eyes-pit eyes-pinholes-spherical lenses-multiple lenses-corneal refraction

-composite eyes-apposition-neural superposition-refracting superposition-reflecting superposition-parabolic superposition

•Other designs match lower computational capabilities, different survival tasks, environment priors


modified optics

3

Computational photography is a holistic approach at solving imaging problems by jointly designing the camera and the signal processing algorithms

Computational photography paradigm


modified optics

3



blurred/coded image


modified optics

3



blurred/coded image

sharp image

blind deconvolution


Example: Coded aperture

4

LCD opaque mask



4

coded image

LCD opaque mask



4

restored image

LCD opaque mask



4

restored image

LCD opaque mask



4

restored image

LCD opaque mask



4

restored image

LCD opaque mask


In this presentation

•The (portable) light field camera

•What can one do with it?•Obtain 3D from a single image•Extend the depth of field•Image synthesis (e.g., digital refocusing)•3D image editing

•How does it work?•Assembly•Camera vs microlens array•Depth estimation•Image deblurring

•What are its limits?•Sampling•Sample repetitions, microlens blur and magnification•Coincidence of samples and undersampling

•Comparisons & evaluation

5


The light-field camera: What can one do with it?

6



6



6


Depth estimation

light field 3D reconstruction

7


Extended depth of field

extended depth of fieldlight field

8


Digital refocusing

9


Digital refocusing

9


Challenge: Limited resolution

10

captured light field



10


4000p

4000p



10


digitally refocused image

4000p

4000p



10


digitally refocused image

4000p

4000p

300p

300p

~178 fold loss


Related work: Superresolution

•Lumsdaine and Georgiev – Tech report 2008 and ICCP 2009Magnification and averaging of microlens images

•Pros: Computationally efficient and simple

•Cons: No deblurring, no depth estimation

11






11






11


Geometric optics

12


p‘

Optical Axis

vv’z

OO

Main lensMicrolenses Sensor

p

i

Geometric optics

12

p


p‘

Optical Axis

vv’z

OO


p

i

Geometric optics

12

p


p‘

Optical Axis

vv’z

OO


p

i

Geometric optics

12

p


p‘

Optical Axis

vv’z

OO


p

i

Geometric optics

12

p


p‘

Optical Axis

vv’z

OO


p

i

Geometric optics

12

p


p‘

Optical Axis

vv’z

OO


p

i

Geometric optics

12

p


p‘

Optical Axis

vv’z

OO


p

i

Geometric optics

12

QR p


p‘

Optical Axis

vv’z

OO


p

i

Geometric optics

12

QR p


p‘

Optical Axis

vv’z

OO


p

i

Geometric optics

12

QR p


p‘

Optical Axis

vv’z

OO


p

i

Geometric optics

12

QR p


p‘

Optical Axis

vv’z

OO


p

i

Geometric optics

12

QR p


Light fields and the light field camera

13


•The light field is a representation of how light propagates in space


13



•Consider a sphere around an object


13

object



•Consider a sphere around an object•The object scatters light


13

object

illumination

reflected light



•Consider a sphere around an object•The object scatters light•We define an intensity value for each position on the sphere and for each 3D direction


13

object

illumination

reflected light

measured light



•Consider a sphere around an object•The object scatters light•We define an intensity value for each position on the sphere and for each 3D direction•The light field can be described by a 4D function


13

object

illumination

reflected light

measured light

viewpoint (u,v)

incoming ray (x,y)light field parametrization

2D coordinate

2D coordinate



•Consider a sphere around an object•The object scatters light•We define an intensity value for each position on the sphere and for each 3D direction•The light field can be described by a 4D function

•The light field camera projects a (portion of the)4D light field onto a 2D sensor array


13

object

illumination

reflected light

measured light

viewpoint (u,v)

incoming ray (x,y)light field parametrization

2D coordinate

2D coordinate

(x,y)

(u,v)

17 December 2011 NIPS 2011 Machine Learning Meets Computational Photography 14

camera vs microlens array

microlens array viewtarget

(x,y)

(u,v)



microlens array view

camera array view

target

(x,y)

(u,v)

(u,v)

(x,y)



microlens array view

camera array view

target

(x,y)

(u,v)

(u,v)

(x,y)


Light field superresolution

15



15



•Key idea: Make use of redundancy in light field images

15




•Formally, superresolution can be posed as a space-varying blind deconvolution problem

15




•Formally, superresolution can be posed as a space-varying blind deconvolution problem•Introduce piecewise smoothness to estimate the depth map of the scene

15




•Formally, superresolution can be posed as a space-varying blind deconvolution problem•Introduce piecewise smoothness to estimate the depth map of the scene•Introduce texture priors to superresolve scene texture given the depth map

15


A Bayesian approach to superresolution

16

l = Hr + w


light field image


16

l = Hr + w


light field image

PSF


16

l = Hr + w


light field image

sharp imagePSF


16

l = Hr + w


light field image

sharp imagePSF

noise


16

l = Hr + w


light field image

sharp imagePSF

noise


16

l = Hr + w

obtain map estimate: r = arg maxr

p(l|r, Hs)p(r)


light field image

sharp imagePSF

noise


16

l = Hr + w

Hs ← hLI = h

MLh

µL


p(l|r, Hs)p(r)


light field image

sharp imagePSF

noise


16

l = Hr + w

Hs ← hLI = h

MLh

µL


p(l|r, Hs)p(r)

hµLk(i)(θq(i), u) =

� 1πb2(u)

��θq(i) − λ(u)(ck(i) − u)��

2< b(u)

0 otherwise.

hMLk(i)(θq(i), u) =

d2

4πβ2,

��θq(i) ±2b(u)

d(ck(i) − u)

��2

<2β

d

0, otherwise


•Energy minimization•Fidelity term is matching all pairs of views via 2D warps•Regularization is Total Variation

•Minimize via linearized Euler-Lagrange equation

Depth reconstruction via stereo matching

17





17





17

Edata(s) =�

u,u,u

Φ�Vu(u− s(u)∆u)− Vu(u− s(u)∆u)

�





17

Edata(s) =�

u,u,u


�

robust norm





17

Edata(s) =�

u,u,u


�

robust norm

view from the vantage pointu





17

Edata(s) =�

u,u,u


�

robust norm depth/disparity map






17

Edata(s) =�

u,u,u


�

robust norm depth/disparity map 2D shift in view centers






17

Edata(s) =�

u,u,u


�

robust norm depth/disparity map 2D shift in view centers


Aliasing needs to be taken into account!


Boundary texture smoothness constraint

18

2 high-res views (wrong depth assumption)

x border gradients:

partial borders interpolated

borders propagated via linear interpolation

pixels used for x-gradients:

xx

xy

cell boundary


Depth superresolution

19

low-res view hi-res view (wrong depth)hi-res view (this method)

low-res depth map hi-res depth map


Experiments

20

camera array view


Experiments

20

camera array view single view


Experiments

20

camera array view single view

recovered depth map


Superresolution

21

@ ulens resolution @ full sensor resolution(Georgiev’s method)

@ full sensor resolution(this work)


Superresolution

21




Superresolution

21



INVERTED DEPTH OF FIELD!


The light field camera: What are its limits?

22

102 103 104 1050

5

10

15

20

25

30

depth (mm)

blur

radi

us (p

ixel

s)

LF cameraRegular camera

plane in focus (635mm)

max blur disc capped by

microlens diameter

F 80mmF# 3.2


Coordinate system & views

23

p‘Optical

Axis

vv’z

OO

Main lens

Microlenses (scale exagerated) Sensorobject planes

conjugate planesp

i


Coordinate system & views

23

p‘Optical

Axis

vv’z

OO

Main lens

Microlenses (scale exagerated) Sensorobject planes

conjugate planesp

i

consider only conjugate domain


•count how many microlenses fall under the blur B

Repetitions

24

θq

θq

µ

θ1θ2

θv’v-

v’ v

θq

θq

main lens blur B


Coincidence of samples and undersampling

25

θ1

θQ

θ2

q=1

q=Q

q=2

µ

q

q



25

θ1

θQ

θ2

q=1

q=Q

q=2

µ

q

q



25

θ1

θQ

θ2

q=1

q=Q

q=2

µ

q

q



25

θ1

θQ

θ2

q=1

q=Q

q=2

µ

q

q



25

θ1

θQ

θ2

q=1

q=Q

q=2

µ

q

q



25

θ1

θQ

θ2

q=1

q=Q

q=2

µ

q

q



25

θ1

θQ

θ2

q=1

q=Q

q=2

µ

q

q



25

θ1

θQ

θ2

q=1

q=Q

q=2

µ

q

q

at these planes some microlenses share the

same identical samples



26

60 80 100 120 140 160 1800

1

2

3

4

5

6

7

8

9

depth level

ISN

R (d

B)

=0=3e 3=6e 3=1.2e 2=2.5e 2=5e 2=1e 1

image reconstruction (experimental validation)

ISNR = 10 log�

r − r0

r − r

�



26

60 80 100 120 140 160 1800

1

2

3

4

5

6

7

8

9

depth level

ISN

R (d

B)

=0=3e 3=6e 3=1.2e 2=2.5e 2=5e 2=1e 1

Georgiev


ISNR = 10 log�

r − r0

r − r

�



26

60 80 100 120 140 160 1800

1

2

3

4

5

6

7

8

9

depth level

ISN

R (d

B)

=0=3e 3=6e 3=1.2e 2=2.5e 2=5e 2=1e 1

Georgiev


this work

ISNR = 10 log�

r − r0

r − r

�



26

60 80 100 120 140 160 1800

1

2

3

4

5

6

7

8

9

depth level

ISN

R (d

B)

=0=3e 3=6e 3=1.2e 2=2.5e 2=5e 2=1e 1

coincidence of samples

Georgiev


this work

ISNR = 10 log�

r − r0

r − r

�



26

60 80 100 120 140 160 1800

1

2

3

4

5

6

7

8

9

depth level

ISN

R (d

B)

=0=3e 3=6e 3=1.2e 2=2.5e 2=5e 2=1e 1

60 80 100 120 140 160 1800

1

2

3

4

5

6

7

8x 10 4

depth level

Aver

age

L 2 erro

r per

pix

el

2x

4x

8x

16x32x

w=0

w=1.2e 2

w=1e 1


Georgiev


this work

ISNR = 10 log�

r − r0

r − r

�



26

60 80 100 120 140 160 1800

1

2

3

4

5

6

7

8

9

depth level

ISN

R (d

B)

=0=3e 3=6e 3=1.2e 2=2.5e 2=5e 2=1e 1

60 80 100 120 140 160 1800

1

2

3

4

5

6

7

8x 10 4

depth level

Aver

age

L 2 erro

r per

pix

el

2x

4x

8x

16x32x

w=0

w=1.2e 2

w=1e 1


Georgiev

low-res reconstruction


this work

ISNR = 10 log�

r − r0

r − r

�



26

60 80 100 120 140 160 1800

1

2

3

4

5

6

7

8

9

depth level

ISN

R (d

B)

=0=3e 3=6e 3=1.2e 2=2.5e 2=5e 2=1e 1

60 80 100 120 140 160 1800

1

2

3

4

5

6

7

8x 10 4

depth level

Aver

age

L 2 erro

r per

pix

el

2x

4x

8x

16x32x

w=0

w=1.2e 2

w=1e 1


Georgiev



coded aperture (Zhou&Nayar mask)

this work

ISNR = 10 log�

r − r0

r − r

�



26

60 80 100 120 140 160 1800

1

2

3

4

5

6

7

8

9

depth level

ISN

R (d

B)

=0=3e 3=6e 3=1.2e 2=2.5e 2=5e 2=1e 1

60 80 100 120 140 160 1800

1

2

3

4

5

6

7

8x 10 4

depth level

Aver

age

L 2 erro

r per

pix

el

2x

4x

8x

16x32x

w=0

w=1.2e 2

w=1e 1


Georgiev



coded aperture (Zhou&Nayar mask) traditional camera

this work

ISNR = 10 log�

r − r0

r − r

�



26

60 80 100 120 140 160 1800

1

2

3

4

5

6

7

8

9

depth level

ISN

R (d

B)

=0=3e 3=6e 3=1.2e 2=2.5e 2=5e 2=1e 1

60 80 100 120 140 160 1800

1

2

3

4

5

6

7

8x 10 4

depth level

Aver

age

L 2 erro

r per

pix

el

2x

4x

8x

16x32x

w=0

w=1.2e 2

w=1e 1


Georgiev



coded aperture (Zhou&Nayar mask)

light field cameratraditional camera

this work

ISNR = 10 log�

r − r0

r − r

�


Comparison with other EDOF systems

27

IEEE TRANSACTION OF PATTERN RECOGNITION AND MACHINE INTELLIGENCE, VOL. , NO. , MONTH YEAR 12

60 80 100 120 140 160 1800

1

2

3

4

5

6

7

8

9

depth level

ISN

R (d

B)

=0=3e 3=6e 3=1.2e 2=2.5e 2=5e 2=1e 1

60 80 100 120 140 160 1800

1

2

3

4

5

6

7

8x 10 4

depth level

Aver

age

L 2 erro

r per

pix

el

2x

4x

8x

16x32x

w=0

w=1.2e 2

w=1e 1

Fig. 17L2 ERROR RESULTS V.S. OTHER SYSTEMS.. LEFT: RESTORATION PERFORMANCE VERSUS DEPTH OF OUR METHOD AND THE METHOD OF [6] ON THE

SIMULATED LF CAMERA USING OUR CAMERA SETTINGS AND THE BRODATZ “BARK” TEXTURE, WITH INPUT INTENSITY RANGE 0–1. THE ISNR IS

COMPARED FOR SEVERAL DIFFERENT LEVELS OF OBSERVATION NOISE (STANDARD DEVIATION σw ). SOLID LINES SHOW OUR RESTORATION METHOD,DASHED USING THE METHOD OF [6] ON THE SAME DATA. WE HAVE NOT RESTORED DEPTHS WHERE λ < 1 (THERE ARE GAPS IN THE RESTORATION AT

THESE PARTS, SINCE SOME PARTS OF THESE PLANES ARE NOT SAMPLED AT ALL.). RIGHT: PERFORMANCE COMPARISON OF DOF EXTENSION BETWEEN

THE LF CAMERA (THICK LINES), A REGULAR CAMERA (THIN LINES) AND A CODED APERTURE CAMERA (DOTTED LINES). THE CROSSES INDICATE THE

ERROR FROM THE UPSAMPLED INTEGRAL REFOCUSING RESULT ON THE SAME LF DATA. SEE MAIN TEXT IN §VIII-B.2 FOR FURTHER DESCRIPTION.

(a) (b) (c) (d) (e) (f) (g) (h)Fig. 18

RESOLUTION TESTS. THE EXPERIMENT IN FIG. 17 IS REPEATED USING PART OF A RESOLUTION TEST CHART AND ADDITIVE NOISE AT σ = 1.2× 10−2 .COLUMN (A): SIMULATED LIGHT FIELD IMAGE; (B) METHOD OF [6] (USED AS INITIALIZATION); (C) METHOD OF [6] DEBLURRED (ONLY FOR

COMPARISON); (D) INPUT IMAGE RESTORED WITH OUR METHOD; (E) SIMULATED CODED APERTURE IMAGE; (F) DECONVOLVED CA IMAGE; (G)SIMULATED FOCAL SWEEP IMAGE (ACROSS THE WHOLE DEPTH RANGE); (H) DECONVOLVED FOCAL SWEEP IMAGE, USING MID-DEPTH PSF. ROWS, TOP

TO BOTTOM: DEPTH=60,72,80,88. THE PLENOPTIC CAMERA IS SEEN TO OUTPERFORM THE CA AND FOCAL SWEEP SYSTEMS IN TERMS OF REGULARITY

AND CLARITY OF THE SOLUTION AWAY FROM THE MAIN-LENS PLANE IN FOCUS. NOTE ALSO THAT MORE DETAIL IS RECOVERED THROUGH USE OF THE

FULL OBSERVATION MODEL THAN WOULD BE POSSIBLE JUST BY DEBLURRING THE RESULTS IN THE SECOND COLUMN.

light field GeorgievGeorgiev

+ deblurring

our method

coded aperture

input

coded aperture

focus sweep input

focus sweep


Conclusions

•We have analyzed the light field camera

•Sampling patterns

•Limits

•We have introduced algorithms for depth and image estimation from a single light field image

•Based on depth and image priors

•Q: What is the tradeoff between depth identification and image reconstruction?

28

Documents

Portable Light Field Imaging: Extended Depth of Field - webdav