View
215
Download
2
Category
Tags:
Preview:
Citation preview
1
Orientation fields and 3D shape Orientation fields and 3D shape estimationestimationRoland W. FlemingMax Planck Institute
for Biological Cybernetics
2
Cues to 3D ShapeCues to 3D Shape
specularities shading texture
Conventional wisdom: different cues have different physical causes must be processed differently by visual system (‘modules’)
3
specularities shading texture
Goal: Find commonalities between cues.
Cues to 3D ShapeCues to 3D Shape
4
Cues to 3D ShapeCues to 3D Shape
5
Cues to 3D ShapeCues to 3D Shape
Fleming, Torralba, Adelson
Todd and colleagues
Mingolla and Grossberg
Koenderink and van Doorn
Zucker and colleagues
Zaidi and Li
Malik and Rosenholtz
6
It is remarkable that we can recover 3D shape:
No motion No stereo No shading No texture
image consists of nothing more than a distorted reflection of the world surrounding the object
Ideal mirrored surface
Fleming et al. (2004). JOV
Shape from SpecularitiesShape from Specularities
7
As the object moves from scene to scene, the image changes dramatically.
Yet, somehow we are able to recover the 3D shape.
Shape from SpecularitiesShape from Specularities
8
Curvatures determine distortionsCurvatures determine distortions
highly curved
9
Curvatures determine distortionsCurvatures determine distortions
slightlycurved
Anisotropies in surface curvature lead to powerful distortions of the reflected world
10
Interpreting distorted reflectionsInterpreting distorted reflections
11
Orientation fieldsOrientation fields
Ground truth
12
3D shape appears to be conveyed by the continuously varying patterns of orientation across the image of a surface
13
Beyond specularityBeyond specularity
Specular reflectionSpecular reflection Diffuse reflectionDiffuse reflection
14
Differences betweenDifferences betweendiffuse and specular reflectiondiffuse and specular reflection
15
Differences betweenDifferences betweendiffuse and specular reflectiondiffuse and specular reflection
16
Differences betweenDifferences betweendiffuse and specular reflectiondiffuse and specular reflection
17
ShinyShiny
Painted Painted
18
Beyond specularityBeyond specularity
Specular reflectionSpecular reflection Diffuse reflectionDiffuse reflection
19
Latent orientationLatent orientationstructurestructure
20
Orientation fieldsOrientation fieldsin shadingin shading
21
Orientation fieldsOrientation fieldsin shadingin shading
22
Reflectance as IlluminationReflectance as Illumination
a(f) = 1 / f
= 0 = 0.4 = 0.8 = 1.2
= 1.6 = 2.0 = 4.0 = 8.0
23
highly curved
24
slightlycurved
Anisotropies in surface curvature lead to anisotropies in the image.
25
Stability across changesStability across changesin surface reflectancein surface reflectance A parametric space of glossy plastic
materials (using Ward model)
Diffuse Reflectance, dDiffuse Reflectance, d
Sp
ecu
lar
Reflect
an
ce,
sS
pecu
lar
Reflect
an
ce,
s
26
Idea: Experiment 1Idea: Experiment 1
Rationale: measure stability of 3D shape across changes in surface reflectance
Method: gauge figure task? Problem: costly to do full depth reconstruction for
many shapes and materials Solution? Compare sparse gauge measurement? Alternative task?:
locate depth extrema along given raster line (2D task)
27
TextureTexture
Anisotropic compression of texture depends on surface slant
28
TextureTexture
Anisotropic compression of texture depends on surface slant
29
Orientation fieldsOrientation fieldsin texturein texture
30
Orientation fieldsOrientation fieldsin texturein texture
31
Orientation fieldsOrientation fieldsin texturein texture
32
Affine TransformationAffine Transformation
Shear:- does affect first derivatives- does NOT affect second derivatives
33
Shear:- does affect first derivatives- does NOT affect second derivatives
Affine TransformationAffine Transformation
34
Shear:- does affect first derivatives- does NOT affect second derivatives
Affine TransformationAffine Transformation
35
Shear:- does affect first derivatives- does NOT affect second derivatives
Affine TransformationAffine Transformation
36
Shear:- does affect first derivatives- does NOT affect second derivatives
Affine TransformationAffine Transformation
37
Shear:- does affect first derivatives- does NOT affect second derivatives
Affine TransformationAffine Transformation
38
Idea: Experiment 2Idea: Experiment 2
Rationale: use orientation fields to predict misperceptions of 3D shape
Possible methods Gauge figure task?
Matching task: subject adjusts shear of a textured
object until it appears to match the shaded version of the same object
Subject adjusts shear of one oject (shaded or textured) until it appears to match the ‘degree of shear’ of another object? Sounds too strange?
39
Illusory distortionsIllusory distortionsof shapeof shape
Inspired by Todd & Thaler VSS 05
40
Illusory distortionsIllusory distortionsof shapeof shape
Inspired by Todd & Thaler VSS 05
41
Idea: Experiment 3Idea: Experiment 3
Rationale: use orientation fields to predict misperceptions of 3D shape
Possible methods gauge figure task to reconstruct
full 3D shape. Again, this is costly, but perhaps
a few shapes are enough
depth extrema task: locate depth extrema along raster line (this is what Todd and Thaler did). Potentially we could predict the
locus directly from the orientation field
42
Idea: Experiment 3Idea: Experiment 3 Compare small and large changes in orientation field by using texture stretching along
the line of sight Advantage: same infringement of ‘isotropy assumption’, different change in apparent
3D shape
UnstretchedUnstretchedStretched 2:1Stretched 2:1
along line of sightalong line of sight
43
Uses biologically plausible measurements
Orientation selectivity maps in primary visual cortex of tree shrew. After Bosking et al. (1997).
Potential of Potential of Orientation FieldsOrientation Fields
44
No need for visual system to estimate reflectance or illumination explicitly.
Classical shape from shading uses the reflectance map to estimate surface normals from image intensities
Reflectance map is usually unknown and ambiguous
Potential of Potential of Orientation FieldsOrientation Fields
45
Stable across albedo discontinuities.
Breton and Zucker (1996), Huggins and Zucker (2001)
Potential of Potential of Orientation FieldsOrientation Fields
46
Handle improbable combinations of reflectance and illumination.
non-linear intensity transfer function
normal shadingnormal shading ‘‘weird’ shadingweird’ shading
Potential of Potential of Orientation FieldsOrientation Fields
47
We could measures shape estimates with these types of stimuli as well.
non-linear intensity transfer function
normal shadingnormal shading ‘‘weird’ shadingweird’ shading
Link back toLink back toexperiment 1 ?experiment 1 ?
48
May explain how images with no obvious BRDF interpretation nevertheless yield 3D percepts
Potential of Potential of Orientation FieldsOrientation Fields
Ohad Ben-Shahar
49
Converting between cuesConverting between cues
input imageinput image
Todd & Oomes 2004
( )2
Latent shadingLatent shading
50
( )2
Converting between cuesConverting between cues
input imageinput image
Todd & Oomes 2004
Latent shadingLatent shading
51
ConclusionsConclusions
Orientation fields are potentially a very powerful source of information about 3D shape
For the early stages of 3D shape processing, seemingly different cues may have more in common than previously thought
52
Thank youThank youCollaborators
Ted AdelsonAntonio Torralba
Funding
RF supported byDFG FL 624/1-1
53
What still needsWhat still needsto be explained?to be explained?
For Lambertian materials (or blurry illuminations), the reflectance map is so smooth that it is significantly anisotropic.
Therefore shading orientation fields vary considerably with changes in illumination.
sidefront top
54
What still needsWhat still needsto be explained?to be explained?
Surprising prediction: 3D shape should actually be less stable across changes in illumination for diffuse than for specular materials.
We found evidence for changes in 3D shape with changes in illumination Alternative: higher order invariants establish an equivalence between
different orientation fields. Example: joint measures of orientation at different locations.
sidefront top
55
Note analogy to textures of different orientations
Todd et al. (2004)
What still needsWhat still needsto be explained?to be explained?
56
Matte dark grey
Rough metal
Glossy light grey
57
PlasticsPlastics
(a) Mirror (b) Smooth plastic (c) Rough plastic
58
When the world is anisotropicWhen the world is anisotropic
Brushed horizontally Brushed vertically
59
Stability across changesStability across changesin surface reflectancein surface reflectance A parametric space of glossy plastic
materials (using Ward model)
Diffuse Reflectance, dDiffuse Reflectance, d
Sp
ecu
lar
Reflect
an
ce,
sS
pecu
lar
Reflect
an
ce,
s
Recommended