MIT Media Lab Camera Culture Image Destabilization: Programmable Defocus using Lens and Sensor Motion Ankit Mohan, Douglas Lanman, Shinsaku Hiura, Ramesh

MIT Media Lab Camera Culture

Image Destabilization:Programmable Defocus using

Lens and Sensor Motion

Ankit Mohan, Douglas Lanman,Shinsaku Hiura, Ramesh Raskar

MIT Media Lab



Lots of glass; Heavy; Bulky; Expensive

Defocus Blur


Image Destabilization

Lens Sensor

Camera

Static Scene


Image Destabilization

Static Scene

Lens Motion Sensor Motion

Camera


Related Work

[Bae and Durand 2007]

[Hasinoff and Kutulakos 2007]

f8

f/4

f/2

f/1

extrapolated aperture

[Hiura et al. 2009][Vaish et al. 2004]


Laminography

X-Ray Source

X-Ray Sensor

Plane of focus

Related technique: Time Delay and Integration (TDI)

Motion direction

Motion direction


A

BA’

B’

Lens based Focusing

Lens Sensor


A

BA’

B’

Lens based Focusing

Lens Sensor


A

B

B’

A’

Lens

Smaller aperture Smaller defocus blur

Sensor


A

BA’

B’

Pinhole

Pinhole: All In-Focus

Sensor


A

B

A’

B’

Pinhole

Shifting Pinhole

Sensor

vp


A

B

Pinhole

A’

B’

Shifting Pinhole

Sensor

vp


A

B

Pinhole

A’

B’

Shifting Pinhole

Sensor

vp


A

B

Pinhole

A’

B’

Shifting Pinhole

Sensor

vp


B

Pinhole Sensor

A’

B’

Shifting Pinhole

A

da

db ds

tp

vp


A

B

A’

B’

Pinhole

Shifting Pinhole and Sensor

vp

Sensor

vs

da

db ds

Focus Here


A

B

Pinhole

A’

B’vp

Sensor

vs

da

db ds

Focus Here



A

B

Pinhole

A’

B’

vp

Sensor

vs

da

db ds

Focus Here



A

B

Pinhole

A’

B’

vp

Sensor

vs

da

db ds

Focus Here



A Lens in Time!

Analogous to shift and sum basedLight field re-focusing.

Lens Equation:

Virtual Focal Length:

Virtual F-Number:


Our Prototype


Adjusting the Focus Plane

all-in-focus pinhole image



focused in the front using destabilization (10 second exposure)



focused in the middle using destabilization (5 second exposure)



focused in the back using destabilization (10 second exposure)


Adjusting the Virtual Aperture

focused in the middle using destabilization (5mm pinhole translation)


Adjusting the Virtual Aperture

focused in the middle using destabilization (30mm pinhole translation)


Shifting Lens and Sensor Defocus

• Physical vs. synthetic focus• Similar physical and synthetic focus

aperture

sensor

real focus

virtual focus

Defocus Exaggeration



static lens with an f/2.8 aperture



destabilization simulates a reduced f-number


Defocus Invariance

aperture

sensor

real focus

virtual focus

• Differing physical and synthetic focus


* =real PSF virtual PSF overall PSF

Defocus Invariance• Related work

– [Nagahara et al. 2008]– [Cathey and Dowski 1995]– [Levin et al. 2008]

• PSF not depth invariant– only size is depth invariant

• Gaussian special case– depth invariant PSF – inversion is ill-conditioned* =

real PSF virtual PSF overall PSF

[Nagahara et al. 2008]


Defocus Invariance

depth-invariant blur size (horizontal slit + destabilization)


Defocus Invariance

Richardson-Lucy deconvolution result


Tilted Sensor

C

DC′

D′

focus plane aperture plane sensor plane

Scheimpflug intersection

dC

dD

d′Cd′D

CC ddf 111


Tilted Sensor

C

D

vs?


vp

CC ddf 111

dC

dD


Tilted Sensor

C

D

vp

C′ D′

vs


dC

dD

d′Cd′D

s

p

DD

D

CC

C

v

v

dd

d

dd

d


Tuning the PSF

aperture

sensor

real focus

pinhole image (static f/22 aperture)


Tuning the PSF

real focus

aperture

sensor

large aperture image (static f/2.8 aperture)


Tuning the PSF

virtual focus

aperture

sensor

destabilized image using a pinhole (translated f/22 aperture)


Tuning the PSF

aperture

sensor

real focus

virtual focus

destabilized image using a large aperture (translated f/2.8 aperture)


Tuning the PSFreal focus

virtual focus

aperture

sensor

simulated aspheric lens using a vertical slit aperture and destabilization


Extension to 2D Displacements

ellipticallinear circular

“figure 8” hypocycloidal trispiral


Large apertures with tiny lenses?Benefits• No time or light inefficiency

wrt cheap cameras• Exploits unused area around

the lens• Compact design• With near-pinhole apertures

(mobile phones) many possibilities

Limitations• Coordinated mechanical

movement required• Diffraction (due to small aperture)

cannot be eliminated[Zhang and Levoy, tomorrow]

[Our group: augmented LF for wave analysis]

• Scene motion during exposure


Acknowledgements

Grace WooMIT CSAIL

Quinn SmithwickMIT Media Lab

Gabriel TaubinBrown University

Jaewon KimMIT Media Lab

MIT Media Lab: Camera Culture


SLRs with tiny lenses?

• Analysis of space of relative lens/sensor displacement• Destabilization as virtual focusing mechanism• Shallower depth of field than physical aperture• Depth-independent defocus blur size

Documents

MIT Media Lab Camera Culture Image Destabilization: Programmable Defocus using Lens and Sensor Motion Ankit Mohan, Douglas Lanman, Shinsaku Hiura, Ramesh