Shading Models for Illumination and Reflectance Invariant Shape DetectorsPeter Nillius†, Josephine Sullivan† and Antonis Argyros‡

KTH, Sweden† FORTH, Greece‡

OverviewWe present a method to construct robust shape detectorsbased on shading. The detectors consist of a PCA appear-ance model which is analytically computed from the imageformation model.Contributions:

Extension of analytical/model-based PCA to includeshape variationsShape detection proof of concept through the construc-tion and evaluation of a sphere and a cylinder detector.

Shading in Frequency SpaceWe model shading from surfaces with any isotropic BRDFunder any illumination. Using frequency space represen-tations, this shading can be expressed as a linear combina-tion of basis functions.

Ii =∑N

k=1 ckEk(αi, βi),where

ck = Lml b

qop

Lml - Light field coefficientsbqop - BRDF coefficients

Ek(αi, βi) - Reflectance map basis functionsWe use N = 2000 number of basis functions to be able torepresent shading from specular surfaces.

Model-Based PCA of ShadingThe frequency space representation can be used to analyt-ically derive the principal components of the set of imagesof a shape. The variations in the images are defined by thevariations in the illumination and BRDF [4, 2, 3].

New Extension to Include Shape VariationsThe principal components are the eigenvectors of the im-age covariance matrix, ΣI . Using the frequency space rep-resentation this matrix can be computed as

ΣI =∑

s∈S FsVcFTs ps(s)

whereVc - Light field and BRDF covariancesFs - Centered basis functions for shape s

ps(s) - Probability prior for shape s

Eigenfaces from four 3D Meshes0.49 (0.485) 0.18 (0.661) 0.11 (0.768) 0.037 (0.805) 0.025 (0.83) 0.019 (0.849) 0.017 (0.867) 0.015 (0.882)

Human Skin BRDF0.2 (0.202) 0.13 (0.331) 0.069 (0.401) 0.066 (0.467) 0.046 (0.513) 0.036 (0.549) 0.026 (0.575) 0.025 (0.599)

Torrance-Sparrow BRDF Mix

Shape DetectionOverall approach:

Score based on normalized residual varianceRun on scale-pyramid to cope with large scale variationsUse multiple models to cope with large pose changesTrain each model to cope with- small scale and pose changes- illumination changes- BRDF changes

Sphere and Cylinder DetectorsTo evaluate the method we have created detectors for twoshape primitives, spheres and cylinders.

Training for Pose VariationsThe models were trained for micro-scale variations andsub-pixel translations. The eight cylinder models were ad-ditionally trained for micro-rotations.

Micro-ScaleVariations

Sup-PixelTranslations

Micro-Rotations

Training for Lighting VariationsThe lighting variations were modeled with nine HDR illu-mination maps each undergoing all 3D rotations.

Training for BRDF VariationsThe BRDF variation were modeled with Torrance-Sparrow of varying specularity and surface roughness.

Surface roughness: m

0.8 0.6 0.4 0.2 0.1

Spec

ular

ity:

kspec 0.25

0.5

0.75

1

The Resulting Models0.34 (0.336) 0.34 (0.673) 0.1 (0.773) 0.055 (0.828) 0.051 (0.879) 0.022 (0.901) 0.022 (0.923) 0.013 (0.936) 0.013 (0.949) 0.01 (0.959)

Sphere Basis Images0.77 (0.766) 0.14 (0.907) 0.031 (0.937) 0.021 (0.958) 0.014 (0.972) 0.011 (0.984) 0.0057 (0.989) 0.0035 (0.993)

Cylinder Basis Images (one of eight directions)

Examples

Sph

ere

Det

ecto

r

Best

Scor

eBe

stSc

ale

Cyl

inde

rD

etec

tor

Best

Scor

eBe

stSc

ale

Best

Dir

ecti

on

Experiments

Synthetic experimentsThe detectors have been tested on all combinations oflighting, BRDF and pose variations. Over 500 millions im-age patches have been used.

For each image patch the appearance ba-sis is fitted and the variance of the resid-ual is computed.

5 10 15 20 25 30 35 40

10−4

10−3

10−2

10−1

100

Number of principal components

Mea

n no

rmal

ized

resi

dual

var

ianc

e

LambertianTS: m = 0.8, kspec = 0.5TS: m = 0.4, kspec = 0.5TS: m = 0.2, kspec = 0.5TS: m = 0.1, kspec = 0.5TS: m = 0.8, kspec = 1TS: m = 0.4, kspec = 1TS: m = 0.2, kspec = 1TS: m = 0.1, kspec = 1

The BSDS300 data set [1] is used as negative exemplarswhen computing ROC-curves. To summarize these resultswe plot the equal error rates.

5 10 15 20 25 30 35 400

0.05

0.1

0.15

0.2

0.25

0.3

Number of principal components

Fals

e Po

sitiv

e R

ate

= 1 −

True

Pos

itive

Rat

e

LambertianTS: m = 0.8, kspec = 0.5TS: m = 0.4, kspec = 0.5TS: m = 0.2, kspec = 0.5TS: m = 0.1, kspec = 0.5TS: m = 0.8, kspec = 1TS: m = 0.4, kspec = 1TS: m = 0.2, kspec = 1TS: m = 0.1, kspec = 1

Sphere Detector

2 4 6 8 10 12 14 16 18 200

0.02

0.04

0.06

0.08

0.1

0.12

0.14

0.16

0.18

0.2

Number of principal components

Fals

e Po

sitiv

e R

ate

= 1 −

True

Pos

itive

Rat

e

LambertianTS: m = 0.8, kspec = 0.5TS: m = 0.4, kspec = 0.5TS: m = 0.2, kspec = 0.5TS: m = 0.1, kspec = 0.5TS: m = 0.8, kspec = 1TS: m = 0.4, kspec = 1TS: m = 0.2, kspec = 1TS: m = 0.1, kspec = 1

Cylinder DetectorThe bases increasingly fit the shapes better than the nega-tive exemplars with an increasing number of components.

Experiments on real imagesWe have created a dataset of 31 images containing threespheres, one paper, one silver painted and one goldpainted which is more specular. The spheres are pho-tographed in a number of different lighting conditions andscales.

In comparison with BSDS300 the spheres fit the model bet-ter. More components leads to better discrimination.

5 10 15 20 25 30 35 40

10−4

10−3

10−2

10−1

Number of principal components

Mea

n no

rmal

ized

resi

dual

var

ianc

e

paper spheresilver spheregold sphereLambertianTS: m = 0.4, kspec = 0.5TS: m = 0.2, kspec = 0.25TS: m = 0.2, kspec = 0.5

Residual Variance

5 10 15 20 25 30 35 400

0.02

0.04

0.06

0.08

0.1

0.12

0.14

0.16

Number of principal components

Fals

e Po

sitiv

e R

ate

= 1 −

True

Pos

itive

Rat

e

paper spheresilver spheregold sphereLambertianTS: m = 0.4, kspec = 0.5TS: m = 0.2, kspec = 0.25TS: m = 0.2, kspec = 0.5

Equal Error

References[1] D. Martin, C. Fowlkes, D. Tal, and J. Malik. A database of human seg-

mented natural images and its appl. to evaluating segmentation algo-rithms and measuring ecological statistics. In ICCV, volume 2, pages416–423, July 2007.

[2] P. Nillius and J. Eklundh. Low-dimensional representations of shadedsurfaces under varying illumination. In CVPR, pages II:185–192, 2003.

[3] P. Nillius and J. Eklundh. Phenomenological eigenfunctions for imageirradiance. In ICCV, pages 568–575, 2003.

[4] R. Ramamoorthi. Analytic pca construction for theoretical analysisof lighting variability in images of a lambertian object. IEEE PAMI,24(10):1322–1333, October 2002.

1