Bayesian Depth-from-Defocus with Shading Constraints

Aaron Karperpaper by

Chen Lin Shuochen Su Yasuyuki Matsushita Kun Zhou Stephen Lin

Dec 17th 2013paper: 2013

Aaron Karper paper by Chen Lin, Shuochen Su, Yasuyuki Matsushita, Kun Zhou, Stephen LinBayesian Depth-from-Defocus with Shading Constraints Dec 17th 2013paper: 2013 1 / 16

1 Overview of Depth from DefocusMAP estimate

2 Overview of Depth from Shading

3 Optimizing both ModelsDepth-from-defocus

4 Results

5 Discussion

Aaron Karper paper by Chen Lin, Shuochen Su, Yasuyuki Matsushita, Kun Zhou, Stephen LinBayesian Depth-from-Defocus with Shading Constraints Dec 17th 2013paper: 2013 2 / 16

Overview of Depth from Defocus

Overview of Depth from Defocus

0 = F−1 − d−1 − v−1d In focus

b =Rv

2

∣∣F−1 − d−1 − v−1d

∣∣ spread out of focus

Model Point-spread as Gaussian

blur h(p|σ2) ∝ exT x2σ2

σ = γb → calibrate γ

I2(p) = I1(p) ∗ h(p|σ22 − σ2

1︸ ︷︷ ︸basically d

)

Aaron Karper paper by Chen Lin, Shuochen Su, Yasuyuki Matsushita, Kun Zhou, Stephen LinBayesian Depth-from-Defocus with Shading Constraints Dec 17th 2013paper: 2013 3 / 16

Overview of Depth from Defocus MAP estimate

Overview of Depth from Defocus – MAP estimate

d̂ = arg maxd

P(d|I1, I2)

= arg maxd

P(I1, I2|d)︸ ︷︷ ︸N around deconv

P(d)︸︷︷︸N (c, 1

2λ )

= arg mind

L(I1, I2|d) + L(d)

=∑

p∈pixels

(I1(p) ∗ h(p|d)− I2(p))2

λ∑i,j∈ε

(di − dj)2

Standard Depth from Defocus

Aaron Karper paper by Chen Lin, Shuochen Su, Yasuyuki Matsushita, Kun Zhou, Stephen LinBayesian Depth-from-Defocus with Shading Constraints Dec 17th 2013paper: 2013 4 / 16

Overview of Depth from Defocus MAP estimate

Overview of Depth from Defocus – MAP estimate

d̂ = arg maxd

P(d|I1, I2)

= arg maxd

P(I1, I2|d)︸ ︷︷ ︸N around deconv

P(d)︸︷︷︸N (c, 1

2λ )

= arg mind

L(I1, I2|d) + L(d)

=∑

p∈pixels

(I1(p) ∗ h(p|d)− I2(p))2

λ∑i,j∈ε

(di − dj)2

Standard Depth from Defocus

Aaron Karper paper by Chen Lin, Shuochen Su, Yasuyuki Matsushita, Kun Zhou, Stephen LinBayesian Depth-from-Defocus with Shading Constraints Dec 17th 2013paper: 2013 4 / 16

Overview of Depth from Defocus MAP estimate

Overview of Depth from Defocus – MAP estimate

d̂ = arg maxd

P(d|I1, I2)

= arg maxd

P(I1, I2|d)︸ ︷︷ ︸N around deconv

P(d)︸︷︷︸N (c, 1

2λ )

= arg mind

L(I1, I2|d) + L(d)

=∑

p∈pixels

(I1(p) ∗ h(p|d)− I2(p))2

λ∑i,j∈ε

(di − dj)2

Standard Depth from Defocus

Aaron Karper paper by Chen Lin, Shuochen Su, Yasuyuki Matsushita, Kun Zhou, Stephen LinBayesian Depth-from-Defocus with Shading Constraints Dec 17th 2013paper: 2013 4 / 16

Overview of Depth from Defocus

Overview of Depth from Defocus

Advantages:

Passive perception

Single camera

Advances in lense technology

Precise

Disadvantages:

Lense aperture necessary

Needs texture

Aaron Karper paper by Chen Lin, Shuochen Su, Yasuyuki Matsushita, Kun Zhou, Stephen LinBayesian Depth-from-Defocus with Shading Constraints Dec 17th 2013paper: 2013 5 / 16

Overview of Depth from Shading

Overview of Depth from Shading

s(n) = 〈Vn,n〉M

n: normals

M: illumination

Measure M with Lambertian sphere.

Uniform albedo:

L(d) = λ∑i,j∈ε

⟨(pj − pi ),

ni + nj‖ni + nj‖

⟩If albedo present, remove with depth estimate.

Aaron Karper paper by Chen Lin, Shuochen Su, Yasuyuki Matsushita, Kun Zhou, Stephen LinBayesian Depth-from-Defocus with Shading Constraints Dec 17th 2013paper: 2013 6 / 16

Overview of Depth from Shading

Overview of Depth from Shading

Advantages:

Passive perception

Single camera

Very precise if applicable

Disadvantages:

Texture hinders perception

Calibration necessary

Aaron Karper paper by Chen Lin, Shuochen Su, Yasuyuki Matsushita, Kun Zhou, Stephen LinBayesian Depth-from-Defocus with Shading Constraints Dec 17th 2013paper: 2013 7 / 16

Optimizing both Models Depth-from-defocus

Optimizing both Models – Depth-from-defocus

Cyclic dependencies in L(I1, I|d)

Resolved by magic1

1Markov random fields and loopy belief propagation. Explanation if requested: 15Aaron Karper paper by Chen Lin, Shuochen Su, Yasuyuki Matsushita, Kun Zhou, Stephen LinBayesian Depth-from-Defocus with Shading Constraints Dec 17th 2013paper: 2013 8 / 16

Optimizing both Models

Optimizing both Models

Depth estimate depends on depth-from-defocus, depth-from-shading.

Depth-from-shading profits from depth estimate for albedo removal.

therefore estimation-maximization

Aaron Karper paper by Chen Lin, Shuochen Su, Yasuyuki Matsushita, Kun Zhou, Stephen LinBayesian Depth-from-Defocus with Shading Constraints Dec 17th 2013paper: 2013 9 / 16

Results

Results

Aaron Karper paper by Chen Lin, Shuochen Su, Yasuyuki Matsushita, Kun Zhou, Stephen LinBayesian Depth-from-Defocus with Shading Constraints Dec 17th 2013paper: 2013 10 / 16

Results

Results

Better results than smoothnessprior.

Especially on untextured regions.

As good results even on texturedregions.

Calibration necessary.

Aaron Karper paper by Chen Lin, Shuochen Su, Yasuyuki Matsushita, Kun Zhou, Stephen LinBayesian Depth-from-Defocus with Shading Constraints Dec 17th 2013paper: 2013 11 / 16

Discussion

Discussion

Calibration hinders application.

Not clear why magic2 wasn’t extended to integrate depth from shading,depth from defocus.

Not clear why sensor fusion wasn’t done.

2Markov random fields and loopy belief propagation. Explanation if requested: 15Aaron Karper paper by Chen Lin, Shuochen Su, Yasuyuki Matsushita, Kun Zhou, Stephen LinBayesian Depth-from-Defocus with Shading Constraints Dec 17th 2013paper: 2013 12 / 16

Discussion

Questions?

Aaron Karper paper by Chen Lin, Shuochen Su, Yasuyuki Matsushita, Kun Zhou, Stephen LinBayesian Depth-from-Defocus with Shading Constraints Dec 17th 2013paper: 2013 13 / 16

Discussion

Sensor Fusion

d

depth

I

intensity

S1 = N (d , σ1(I ))

S2 = N (d , σ2(I ))

Estimate global σ for both models.

Use to estimate d for each pixel.

Aaron Karper paper by Chen Lin, Shuochen Su, Yasuyuki Matsushita, Kun Zhou, Stephen LinBayesian Depth-from-Defocus with Shading Constraints Dec 17th 2013paper: 2013 14 / 16

Discussion

Markov random fields

Each node represents a value (a proposition).

Each node has a belief3.

Each node can depend on other nodes.

Connections if dependency.

Basically undirected Bayes Net.

Solve with Belief Propagation.

3Like a probability, but can take values in R≥0Aaron Karper paper by Chen Lin, Shuochen Su, Yasuyuki Matsushita, Kun Zhou, Stephen LinBayesian Depth-from-Defocus with Shading Constraints Dec 17th 2013paper: 2013 15 / 16

Discussion

Belief Propagation

Nodes send own belief to nodes that depend on them.

Update belief on message.

Pray for convergence.

Aaron Karper paper by Chen Lin, Shuochen Su, Yasuyuki Matsushita, Kun Zhou, Stephen LinBayesian Depth-from-Defocus with Shading Constraints Dec 17th 2013paper: 2013 16 / 16