1 Orientation fields and 3D shape estimation Roland W. Fleming Max Planck Institute for Biological Cybernetics

1

Orientation fields and 3D shape Orientation fields and 3D shape estimationestimationRoland W. FlemingMax Planck Institute

for Biological Cybernetics

2

Cues to 3D ShapeCues to 3D Shape

specularities shading texture

Conventional wisdom: different cues have different physical causes must be processed differently by visual system (‘modules’)

3

specularities shading texture

Goal: Find commonalities between cues.


4


5


Fleming, Torralba, Adelson

Todd and colleagues

Mingolla and Grossberg

Koenderink and van Doorn

Zucker and colleagues

Zaidi and Li

Malik and Rosenholtz

6

It is remarkable that we can recover 3D shape:

No motion No stereo No shading No texture

image consists of nothing more than a distorted reflection of the world surrounding the object

Ideal mirrored surface

Fleming et al. (2004). JOV

Shape from SpecularitiesShape from Specularities

7

As the object moves from scene to scene, the image changes dramatically.

Yet, somehow we are able to recover the 3D shape.

Shape from SpecularitiesShape from Specularities

8

Curvatures determine distortionsCurvatures determine distortions

highly curved

9

Curvatures determine distortionsCurvatures determine distortions

slightlycurved

Anisotropies in surface curvature lead to powerful distortions of the reflected world

10

Interpreting distorted reflectionsInterpreting distorted reflections

11

Orientation fieldsOrientation fields

Ground truth

12

3D shape appears to be conveyed by the continuously varying patterns of orientation across the image of a surface

13

Beyond specularityBeyond specularity

Specular reflectionSpecular reflection Diffuse reflectionDiffuse reflection

14

Differences betweenDifferences betweendiffuse and specular reflectiondiffuse and specular reflection

15


16


17

ShinyShiny

Painted Painted

18

Beyond specularityBeyond specularity

Specular reflectionSpecular reflection Diffuse reflectionDiffuse reflection

19

Latent orientationLatent orientationstructurestructure

20

Orientation fieldsOrientation fieldsin shadingin shading

21

Orientation fieldsOrientation fieldsin shadingin shading

22

Reflectance as IlluminationReflectance as Illumination

a(f) = 1 / f

= 0 = 0.4 = 0.8 = 1.2

= 1.6 = 2.0 = 4.0 = 8.0

23

highly curved

24

slightlycurved

Anisotropies in surface curvature lead to anisotropies in the image.

25

Stability across changesStability across changesin surface reflectancein surface reflectance A parametric space of glossy plastic

materials (using Ward model)

Diffuse Reflectance, dDiffuse Reflectance, d

Sp

ecu

lar

Reflect

an

ce,

sS

pecu

lar

Reflect

an

ce,

s

26

Idea: Experiment 1Idea: Experiment 1

Rationale: measure stability of 3D shape across changes in surface reflectance

Method: gauge figure task? Problem: costly to do full depth reconstruction for

many shapes and materials Solution? Compare sparse gauge measurement? Alternative task?:

locate depth extrema along given raster line (2D task)

27

TextureTexture

Anisotropic compression of texture depends on surface slant

28

TextureTexture

Anisotropic compression of texture depends on surface slant

29

Orientation fieldsOrientation fieldsin texturein texture

30


31


32

Affine TransformationAffine Transformation

Shear:- does affect first derivatives- does NOT affect second derivatives

33



34



35



36



37



38


Rationale: use orientation fields to predict misperceptions of 3D shape

Possible methods Gauge figure task?

Matching task: subject adjusts shear of a textured

object until it appears to match the shaded version of the same object

Subject adjusts shear of one oject (shaded or textured) until it appears to match the ‘degree of shear’ of another object? Sounds too strange?

39

Illusory distortionsIllusory distortionsof shapeof shape

Inspired by Todd & Thaler VSS 05

40

Illusory distortionsIllusory distortionsof shapeof shape

Inspired by Todd & Thaler VSS 05

41


Rationale: use orientation fields to predict misperceptions of 3D shape

Possible methods gauge figure task to reconstruct

full 3D shape. Again, this is costly, but perhaps

a few shapes are enough

depth extrema task: locate depth extrema along raster line (this is what Todd and Thaler did). Potentially we could predict the

locus directly from the orientation field

42

Idea: Experiment 3Idea: Experiment 3 Compare small and large changes in orientation field by using texture stretching along

the line of sight Advantage: same infringement of ‘isotropy assumption’, different change in apparent

3D shape

UnstretchedUnstretchedStretched 2:1Stretched 2:1

along line of sightalong line of sight

43

Uses biologically plausible measurements

Orientation selectivity maps in primary visual cortex of tree shrew. After Bosking et al. (1997).

Potential of Potential of Orientation FieldsOrientation Fields

44

No need for visual system to estimate reflectance or illumination explicitly.

Classical shape from shading uses the reflectance map to estimate surface normals from image intensities

Reflectance map is usually unknown and ambiguous


45

Stable across albedo discontinuities.

Breton and Zucker (1996), Huggins and Zucker (2001)


46

Handle improbable combinations of reflectance and illumination.

non-linear intensity transfer function

normal shadingnormal shading ‘‘weird’ shadingweird’ shading


47

We could measures shape estimates with these types of stimuli as well.

non-linear intensity transfer function

normal shadingnormal shading ‘‘weird’ shadingweird’ shading

Link back toLink back toexperiment 1 ?experiment 1 ?

48

May explain how images with no obvious BRDF interpretation nevertheless yield 3D percepts


Ohad Ben-Shahar

49

Converting between cuesConverting between cues

input imageinput image

Todd & Oomes 2004

( )2

Latent shadingLatent shading

50

( )2

Converting between cuesConverting between cues

input imageinput image

Todd & Oomes 2004

Latent shadingLatent shading

51

ConclusionsConclusions

Orientation fields are potentially a very powerful source of information about 3D shape

For the early stages of 3D shape processing, seemingly different cues may have more in common than previously thought

52

Thank youThank youCollaborators

Ted AdelsonAntonio Torralba

Funding

RF supported byDFG FL 624/1-1

53

What still needsWhat still needsto be explained?to be explained?

For Lambertian materials (or blurry illuminations), the reflectance map is so smooth that it is significantly anisotropic.

Therefore shading orientation fields vary considerably with changes in illumination.

sidefront top

54


Surprising prediction: 3D shape should actually be less stable across changes in illumination for diffuse than for specular materials.

We found evidence for changes in 3D shape with changes in illumination Alternative: higher order invariants establish an equivalence between

different orientation fields. Example: joint measures of orientation at different locations.

sidefront top

55

Note analogy to textures of different orientations

Todd et al. (2004)


56

Matte dark grey

Rough metal

Glossy light grey

57

PlasticsPlastics

(a) Mirror (b) Smooth plastic (c) Rough plastic

58

When the world is anisotropicWhen the world is anisotropic

Brushed horizontally Brushed vertically

59

Stability across changesStability across changesin surface reflectancein surface reflectance A parametric space of glossy plastic

materials (using Ward model)

Diffuse Reflectance, dDiffuse Reflectance, d

Sp

ecu

lar

Reflect

an

ce,

sS

pecu

lar

Reflect

an

ce,

s

Documents

1 Orientation fields and 3D shape estimation Roland W. Fleming Max Planck Institute for Biological Cybernetics