Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts

Image Statistics and the Image Statistics and the Perception of 3D ShapePerception of 3D ShapeRoland W. FlemingMax Planck Institute

for Biological Cybernetics

Yuanzhen Li

Edward H. AdelsonMassachusetts Institute of Technology

Matte

Glossy

Mirrored

Hen

ry M

oore

Visual system estimates surface orientation from image intensity

Classical Classical Shape from ShadingShape from Shading

reflectance mapimage



reflectance map

Problems: Intensities are ambiguous

Reflectance map is unknown

No principled way to predict successes vs. failures of shape perception

Surface reflectanceSurface reflectance

A parametric space of glossy plastic materials (using Ward model)

Diffuse Reflectance, dDiffuse Reflectance, d

Sp

ecu

lar

Reflect

an

ce,

sS

pecu

lar

Reflect

an

ce,

s

Don’t use image intensity ! Use the kinds of image measurements the visual

system employs at the front end

Alternative approachAlternative approach


Don’t use image intensity ! Use the kinds of image measurements the visual

system employs at the front end

Alternative approachAlternative approach

image

What can these measurements tell us about 3D shape ?

Can filter responses predict human shape perception ?

highly curved

Curvatures determineCurvatures determinedistortionsdistortions

slightlycurved

Anisotropies in surface curvature lead to powerful distortions of the reflected world

Curvatures determineCurvatures determinedistortionsdistortions

Population codesPopulation codes




StatisticsStatisticsIlluminations

Sh

ap

es

Render many images: 50 Shapes 12 Illuminations 5 Reflectances

Measure the distribution of orientations (i.e. filter population response) for every point in every image

Look for regularities

Orientation fieldsOrientation fields

Ground truth


Error (estimate - ground truth)

Surface reflectanceSurface reflectance

Diffuse Reflectance, dDiffuse Reflectance, d

Sp

ecu

lar

Reflect

an

ce,

sS

pecu

lar

Reflect

an

ce,

s

If the visual system relies on these measurements then:

1: Shape perception should be stable across changes that do not affect these measurements

2: Perceived shape should vary systematically when scene or image modifications do affect these measurements

PredictionsPredictions

Perceived shape should be extremely stable across changes in surface glossiness.

Prediction 1Prediction 1

Nefs, Koenderink & Kappers, 2006

“We found no evidence that the perceived shapes of glossy objects are different from the perceived shapes of matte objects...”

ExperimentExperiment

Specular reflectionSpecular reflection Diffuse reflectionDiffuse reflection

ExperimentExperiment



ground truth


ground truth

For shaded surfaces, perceived shape should undergo (subtle) changes across variations in illumination






Todd, Norman, Koenderink & Kappers (1997) report little effect of illumination. But that was with additional cues.

Koenderink, van Doorn, Christou & Lappin (1996)

Nefs, Koenderink & Kappers, 2006



Caniard & Fleming, 2007


1: Shape perception should be stable across changes that do not affect these measurements. Even when these changes are not natural.

2: Perceived shape should vary systematically when scene or image modifications do affect these measurements


Test improbable combination of lighting and reflectance

Decouple intensity from image orientation

non-linear intensity transfer function

normal shadingnormal shading ‘‘weird’ shadingweird’ shading

““Weird” ShadingWeird” Shading

normal shadingnormal shading‘‘weird’ shadingweird’ shading

““Weird” shadingWeird” shading



perceived tiltperceived tilt

perceived slantperceived slant

normal shadingnormal shading

““ weir

d”

shadin

gw

eir

d”

shad

ing

S1

S1



perceived tiltperceived tilt

perceived slantperceived slant


““ weir

d”

shadin

gw

eir

d”

shad

ing

S2

S2


‘‘ weir

d’

shad

ing

weir

d’

shad

ing


Pooled data across 6 shapes

tilttilt slantslant


r2 = 0.93 r2 = 0.88

Affine TransformationAffine Transformation

Shear:- does affect first derivatives- does NOT affect second derivatives







Matching TaskMatching Task

Subject adjusts shear of match until it appears to be same shape as test

test match



test match



test match



test match


test shear

matc

h s

hear ve

ridi

cal

image statisticsprediction

ResultsResults

test shear

matc

h s

hear


1: Shape perception should be stable across changes that do not affect these measurements.

2: Perceived shape should vary systematically when scene or image modifications do affect these measurements. Even when these changes are not natural.


Illusory distortionsIllusory distortionsof shapeof shape

Inspired by Todd & Thaler VSS 05







Experiment

ResultsResultsveridicalstimulus

ResultsResultspredictedstimulus

ResultsResultsresultsstimulus

Dot product between subject’s data and predictions

ResultsResults


ResultsResults


ResultsResults

“veridical”prediction

“ori

en

tati

on

field

”p

red

icti

on


ResultsResults

“veridical”prediction

“ori

en

tati

on

field

”p

red

icti

on

ConclusionsConclusions

Useful shape cues can be derived from relatively simple image measurements at the front end of vision

In some cases these measurements are surprisingly robust across variations in other scene properties (e.g. illumination, reflectance).

Scale and orientation measurements can predict certain successes and failures of human 3D shape perception across a range of natural and unnatural stimuli.

Thank youThank youFunding

RF supported byDFG FL 624/1-1

Generative space of all possible combinations of surface curvature and local orientation in the reflectance map

Expected errorsExpected errors

Reflectance as IlluminationReflectance as Illumination

a(f) = 1 / f

= 0 = 0.4 = 0.8 = 1.2

= 1.6 = 2.0 = 4.0 = 8.0

Cues to 3D ShapeCues to 3D Shape

specularities shading texture

Conventional wisdom: different cues have different physical causes must be processed differently by visual system (‘modules’)

specularities shading texture

Goal: Find commonalities between cues.




Fleming, Torralba, Adelson

Todd and colleagues

Mingolla and Grossberg

Koenderink and van Doorn

Zucker and colleagues

Zaidi and Li

Malik and Rosenholtz

Rendering withRendering withReflectance mapsReflectance maps

Reflectance map is a lookup-table that specifies image intensity for all surface normals Surface normals are indices for accessing values from the reflectance map

Within a local patch of surface, the normal changes smoothly This maps a small patch of the reflectance map “texture” into the image The rate at which the indices sweep through the reflectance map determines the warping

transformation that is applied to the texture patch during the mapping

Hierarchy of shapeHierarchy of shapeattributesattributes

We often refer to “stereo” or “texture”, or “shading” as “cues” to shape.

Traditional definition of shape cue: a physical property that can inform us about shape, e.g. “stereo”, or “texture”, or “shading”

New definition of cue: a specific image measurement that provides statistically reliable information about a specific property of the scene.

Any given cue on its own may be highly ambiguous, specifying some abstract, high level scene property that does not uniquely specify the object

Hierarchy of shapeHierarchy of shapeattributesattributes

Easily measurable image statistics that can inform us about any property of shape

It is remarkable that we can recover 3D shape:

No motion No stereo No shading No texture

image consists of nothing more than a distorted reflection of the world surrounding the object

Ideal mirrored surface

Fleming et al. (2004). JOV

Shape from SpecularitiesShape from Specularities

As the object moves from scene to scene, the image changes dramatically.

Yet, somehow we are able to recover the 3D shape.


Approach IApproach I::inverse opticsinverse optics

Estimate shape by inverting the physics of mirror reflections.

Image from Savarese and Perona

Make an explicit model of the environment

Make assumptions about specific environmental features (e.g. ‘lines are straight’)

Estimate shape directly from the image Collect image measurements that are reliable

across ‘typical’ environments

Approach IIApproach II::direct perceptiondirect perception

No need to estimate the environment

ust use the pattern of distortions in the image

Pattern of compressions and rarefactions across the image indicates something about the 3D shape.

Shape from TextureShape from Texture

Real-world illumination is highly structured Specular reflections of the real world are a bit like texture Can we solve the 3D shape of mirrors using shape-from-

texture ?

Shape from TextureShape from Texture

??

Slant distorts texture but not reflections

Image distortionsImage distortions



Curvature distorts reflections but not texture


Shape-from-textureShape-from-textureandand

shape-from-specularityshape-from-specularityfollow different rulesfollow different rules

For texture, image compression depends on surface slant

first derivative of surface

For reflections, image compression depends on surface curvature properties

second derivatives of surface

Local analysis: Local analysis: banding patternsbanding patterns

Gauge Figure TaskGauge Figure Task

Subject adjusts 3D orientation of “gauge figure” to match local orientation of surface

Slant and TiltSlant and Tilt

Image from Palmer, 1999

Results IResults I

objective tilt

subje

ctiv

e t

ilt

TiltTilt

objective slant

subje

ctiv

e s

lant

SlantSlant

objective tilt

subje

ctiv

e t

ilt

objective slant

subje

ctiv

e s

lant

TiltTilt SlantSlant

Results IIResults II

objective tilt

subje

ctiv

e t

ilt

TiltTilt

objective slant

subje

ctiv

e s

lant

SlantSlant

objective tilt

subje

ctiv

e t

ilt

objective slant

subje

ctiv

e s

lant

TiltTilt SlantSlant

Is it just the occluding contour?Is it just the occluding contour?

No, it is not

Interpreting distortedInterpreting distortedreflectionsreflections

Effects of Effects of compressioncompression

3D shape appears to be conveyed by the continuously varying patterns of orientation across the image of a surface

Beyond specularityBeyond specularity


Differences betweenDifferences betweendiffuse and specular reflectiondiffuse and specular reflection



ShinyShiny

Painted Painted

Beyond specularityBeyond specularity


Latent orientationLatent orientationstructurestructure

Orientation fieldsOrientation fieldsin shadingin shading

Orientation fieldsOrientation fieldsin shadingin shading

highly curved

slightlycurved

Anisotropies in surface curvature lead to anisotropies in the image.

TextureTexture

Anisotropic compression of texture depends on surface slant

TextureTexture

Anisotropic compression of texture depends on surface slant

Orientation fieldsOrientation fieldsin texturein texture



No need for visual system to estimate reflectance or illumination explicitly.

Classical shape from shading uses the reflectance map to estimate surface normals from image intensities

Reflectance map is usually unknown and ambiguous

Potential of Potential of Orientation FieldsOrientation Fields




Stable across albedo discontinuities.

Breton and Zucker (1996), Huggins and Zucker (2001)


Uses biologically plausible measurements

Orientation selectivity maps in primary visual cortex of tree shrew. After Bosking et al. (1997).


May explain how images with no obvious BRDF interpretation nevertheless yield 3D percepts


Ohad Ben-Shahar

Converting between cuesConverting between cues

input imageinput image

Todd & Oomes 2004

( )2

Latent shadingLatent shading

( )2

Converting between cuesConverting between cues

input imageinput image

Todd & Oomes 2004

Latent shadingLatent shading

Matte vs. ShinyMatte vs. Shiny Same generative statistics, different mappings

Mapped Mapped as as texturetexture

Mapped as Mapped as reflectionreflection

Mapped Mapped as as texturetexture

Mapped as Mapped as reflectionreflection

Texture vs. ReflectanceTexture vs. Reflectance




ConclusionsConclusions

Orientation fields are potentially a very powerful source of information about 3D shape

For the early stages of 3D shape processing, seemingly different cues may have more in common than previously thought

Todd’s BlobsTodd’s Blobs

Todd’s BlobsTodd’s Blobs

What still needs to be explained?What still needs to be explained?

For Lambertian materials (or blurry illuminations), the reflectance map is so smooth that it is significantly anisotropic.

Therefore shading orientation fields vary considerably with changes in illumination.

sidefront top

What still needs to be explained?What still needs to be explained?

Note analogy to textures of different orientations

Todd et al. (2004)

Two possibilitiesTwo possibilities

I. Change in orientation field predicts (subtle) changes in perceived 3D shape

II. There are higher-order invariants in the orientation fields

sidefront top

Eigenvectors of Hessian matrix

Intrinsic principal curvatures

Matte dark grey

Rough metal

Glossy light grey

PlasticsPlastics

(a) Mirror (b) Smooth plastic (c) Rough plastic

When the world is anisotropicWhen the world is anisotropic

Brushed horizontally Brushed vertically

Stripy worldStripy world

Matte vs. ShinyMatte vs. Shiny Same generative statistics, different mappings

Mapped as texture

Mapped as reflection

Mapped as texture

Mapped as reflection

Hypothesis: the way the reflections are distorted is systematically related to properties of the 3D shape


Documents

Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts