EE 576 - Shape Descriptorsisl.ee.boun.edu.tr/courses/ee576/lectures/sunum/shaped...Selects N data items at random Estimates parameter Finds how many data items (of M) ﬁt the model

Outline

Shape Descriptors

Edges

EE 576 - Shape Descriptors

H.I. Bozma

Electric Electronic Engineering

Bogazici University

April 22, 2020

H.I. Bozma EE 576 - Shape Descriptors

Outline

Shape Descriptors

Edges

Shape Descriptors2D Shape DescriptionGeometric PropertiesPrincipal Component Analysis

EdgesParameter EstimationRANSACBounding BoxElliptic Fourier DescriptorsShape from Shading


Outline

Shape Descriptors

Edges

2D Shape Description

Geometric Properties

Principal Component Analysis

Shape Descriptors


Outline

Shape Descriptors

Edges




What to look for?

Good shape descriptors can reduce complexity of recognition.

◮ Stable: Small changes in the data → Small changes in therepresentation

◮ Rich: Ability to describe differences and similarities

◮ Occlusion: In case of occlusion, change should be partial


Outline

Shape Descriptors

Edges




2D Models

Some most commonly used region based shape descriptors are:

◮ Segments and geometric properties (area,moments(elongation), profiles, orientation)

◮ Skeleton

◮ Generalized cylinders

Some most commonly boundary based shape descriptors are:

◮ Geometric properties

◮ Bounding Boxes

◮ 2D Elliptic Fourier Descriptors


Outline

Shape Descriptors

Edges




Analysis for different features of the underlying image objects.Area A – 0th moment of the object:

A =

∫

b(x)dx =

∫ ∫

b(x1, x2)dx1dx2 (1)

The center of the mass - First moment

x =

∫

xb(x)dx

Principal axes - Second moments

x =

∫

(x − x)(x − x)Tb(x)dx


Outline

Shape Descriptors

Edges




Moments


Outline

Shape Descriptors

Edges




PCA

Principal Component Analysis (PCA) - The computation ofprincipal axes of a binary object – which is basically a cluster ofpoints.

◮ Finding the eigenvalues λi ,i = 1, 2 and eigenvectors of the M2

matrix

◮ The eigenvectors are orthonormal vectors

◮ Construct the rotation matrix for coordinate alignment

◮ Let λ1 > λ2 wlog◮ If λ1 = λ2 → Symmetric object◮ If λ1 > λ2 = 0 → Binary object looks like a line


Outline

Shape Descriptors

Edges

Parameter Estimation

RANSAC

Bounding Box

Elliptic Fourier Descriptors

Shape from Shading

Line Fitting

The axis of minimum inertia – identify orientation.Fitting a line the binary object and estimating its parameters.

l tx + α3 = 0 (2)

where

l =

∣

∣

∣

∣

α1

α2

∣

∣

∣

∣

(3)

or equivalent in homogeneous coordinates

l t x = 0 (4)

where

l =

∣

∣

∣

∣

∣

∣

α1

α2

α3

∣

∣

∣

∣

∣

∣

(5)


Outline

Shape Descriptors

Edges


RANSAC

Bounding Box


Shape from Shading


Use two parameters θ and ρ.

◮ ρ : Distance of the line from the origin of the coordinatesystem,

◮ θ: Angle between the line and the x1-axis and

◮ r is the shortest distance to the line (the perpendiculardistance):

l =

∣

∣

∣

∣

sin θ−cos θ

∣

∣

∣

∣

(6)


Outline

Shape Descriptors

Edges


RANSAC

Bounding Box


Shape from Shading

Orientation: Alternative Approach

Figure: Line parametrization


Outline

Shape Descriptors

Edges


RANSAC

Bounding Box


Shape from Shading

Orientation (cont.)

The best-fit line should minimize:

I =

∫

r2b(x)dx

r - Distance between a point x and the closest point x0 on the line

xl =

∣

∣

∣

∣

−ρ sin θ + tcos θ

ρcos θ + t sin θ

∣

∣

∣

∣

t - Distance along the line from the closest point xl to the origin.

r2 = (x − x0)T (x − x0)

= x21 + x22 + ρ2 + 2ρ(x1sinθ − x2cosθ)− 2t(x1cosθ + x2sinθ) + t2


Outline

Shape Descriptors

Edges


RANSAC

Bounding Box


Shape from Shading

Orientation (cont.)

For finding the minimal t, differentiating wrt to t,

t = x1cos θ + x2 sin θ (7)

Substituting into x − xl ,

r2 = (x1sinθ − x2cosθ + ρ)2 (8)


Outline

Shape Descriptors

Edges


RANSAC

Bounding Box


Shape from Shading

Orientation (cont.)

In order to find the minimal I , take partial derivative wrt to ρ:

∫

2(x1sin θ − x2cos θ + ρ)(x)b(x)dx = 0

Multiplying and dividing by A =∫

b(x)dx ,

A(x1sin θ − x2cos θ + ρ) = 0

which can be solved to find ρ.


Outline

Shape Descriptors

Edges


RANSAC

Bounding Box


Shape from Shading

Orientation (cont.)

Next, apply the following simple change of coordinates

x = x − x

Hence, the line equation becomes:

xT l + ρ = xT l

Hence, where

l =

∣

∣

∣

∣

sin

−cos θ

∣

∣

∣

∣


Outline

Shape Descriptors

Edges


RANSAC

Bounding Box


Shape from Shading

Orientation (cont.)

I =

∫

r2b(x)dx

= (xT l)2b(x)dx

= a1sin2θ − a2sin θcos θ + a3cos

2θ

where

a1 =

∫

x21b(x)dx

a2 =

∫

x1x2b(x)dx

a3 =

∫

x22b(x)dx


Outline

Shape Descriptors

Edges


RANSAC

Bounding Box


Shape from Shading

Orientation (cont.)

Using trigonometric identities,

I =1

2(a1 + a3)−

1

2(a1 − a3)cos2θ −

1

2a2sin2θ


Outline

Shape Descriptors

Edges


RANSAC

Bounding Box


Shape from Shading

Finding Angle

Differentiating I wrt to θ and setting the result to zero,

tan2θ =a2

a1 − a3

unless a2 = 0 and a1 = a3.


Outline

Shape Descriptors

Edges


RANSAC

Bounding Box


Shape from Shading

Finding Angle

Two solutions exist:

◮ Positive solution → Orientation of major principal axis →Minimizing I

◮ Negative solution → Orientation of the minor principal axis →Maximizing I

◮Imin

Imax→ How rounded the object is.

◮ Line: Imin

Imax= 0

◮ Circle: Imin

Imax= 1


Outline

Shape Descriptors

Edges


RANSAC

Bounding Box


Shape from Shading

Random Sample Consensus

Assume:

1. The parameters can be estimated from N data items.

2. There are M data items in total.

3. The probability of a randomly selected data item being part ofa good model

4. The probability that the algorithm will exit without finding agood fit if one exists


Outline

Shape Descriptors

Edges


RANSAC

Bounding Box


Shape from Shading

Algorithm

◮ Selects N data items at random

◮ Estimates parameter

◮ Finds how many data items (of M) fit the model withparameter vector within a user given tolerance. Call this K.

◮ If K is big enough, accept fit and exit with success.

◮ Repeat 1-4 L times where L is computed based on prob.

◮ Fail if you get here


Outline

Shape Descriptors

Edges


RANSAC

Bounding Box


Shape from Shading

Skeleton

• The skeleton of a region – the medial axis transform:Defined as follows for a region R with border δR :

◮ For each point x in R, find closest neighbour in δR .

◮ If x has more than one such closest neighbour, then x belongsto the medial axis (or skeleton) of R.

◮ The closest is defined by the metric.


Outline

Shape Descriptors

Edges


RANSAC

Bounding Box


Shape from Shading

Skeleton

Figure: Skeletons using Euclidean metric.


Outline

Shape Descriptors

Edges


RANSAC

Bounding Box


Shape from Shading

Medial-Axis Transform - Morphological operators

Sk(R) = (R ⊖ kB)− [(R ⊖ kB) ◦ B] (9)

Note that if B is a structuring element, (R ⊖ kB) indicates ksuccessive erosions of A.Let K denote the last iterative step before R erodes to ∅.

S(R) = ∪k=0K Sk(R)


Outline

Shape Descriptors

Edges


RANSAC

Bounding Box


Shape from Shading

Medial-Axis Transform

Region R can be reconstructed from its skeleton subsets:

R = ∪k=0K (Sk(R)⊕ kB)

(A⊕ kB) indicates k successive dilations of A.


Outline

Shape Descriptors

Edges


RANSAC

Bounding Box


Shape from Shading


Figure: From top to bottom, left to right: Left: Region R , RightStructuring element B .


Outline

Shape Descriptors

Edges


RANSAC

Bounding Box


Shape from Shading


Figure: From top to bottom, left to right: a) R ⊖B , b) R ⊖ 2B , c) R ◦B .


Outline

Shape Descriptors

Edges


RANSAC

Bounding Box


Shape from Shading


Figure: From top to bottom, left to right: d) (R ⊖ B) ◦ B , e)S0 = R − R ◦ B , f) S1 = (R ⊖ B)− (R ⊖ B) ◦ B .


Outline

Shape Descriptors

Edges


RANSAC

Bounding Box


Shape from Shading


Figure: From top to bottom, left to right: g)S2 = (R − 2B)− (R − 2B) ◦ B , h) S = S0 ∪ S1 ∪ S2 .


Outline

Shape Descriptors

Edges


RANSAC

Bounding Box


Shape from Shading



Outline

Shape Descriptors

Edges


RANSAC

Bounding Box


Shape from Shading

Generalized Cylinders

Generalized cylinders can be modeled as follows:

◮ Skeleton: First defining a parametric curve that acts as theaxis of the cylinder

◮ Then defining a cross section that is swept along the axis.


Outline

Shape Descriptors

Edges


RANSAC

Bounding Box


Shape from Shading


◮ The segments of the curve are represented by Hermite curveswhich are defined by two control points and a tangent vectorat each control point.

◮ Defining the axis of a generalized cylinder → Defining a set oftangent vectors and control points.

◮ This set will define a parametric curve that is comprised of asequence of cubic curve segments.


Outline

Shape Descriptors

Edges


RANSAC

Bounding Box


Shape from Shading



Outline

Shape Descriptors

Edges


RANSAC

Bounding Box


Shape from Shading

Bounding Box

◮ Choose the extremum points

◮ Fit a rectangle to these points


Outline

Shape Descriptors

Edges


RANSAC

Bounding Box


Shape from Shading

Bounding Box

Figure: Bounding box


Outline

Shape Descriptors

Edges


RANSAC

Bounding Box


Shape from Shading


Elliptic Fourier descriptors – Represent as the sum of basisconsisting of sinusoidal functions.

x(t) =

[

a0b0

]

+H∑

i=1

[

ai bici di

] [

cos(iωt)sin(iωt)

]


Outline

Shape Descriptors

Edges


RANSAC

Bounding Box


Shape from Shading


Outline

Shape Descriptors

Edges


RANSAC

Bounding Box


Shape from Shading


ai =1

2π

∫ 2π

0

x1(t) cos(it)dt (10)

bi =1

2π

∫ 2π

0

x1(t) sin(it)dt (11)

ci =1

2π

∫ 2π

0

x2(t) cos(it)dt (12)

di =1

2π

∫ 2π

0

x2(t) sin(it)dt (13)


Outline

Shape Descriptors

Edges


RANSAC

Bounding Box


Shape from Shading

Discrete Elliptic Fourier Descriptors

ak =1

πωk2

P∑

p1

∆xp1∆tp

(cos(kωtp)− cos(kωtp−1))

bk =1

πωk2

P∑

p1

∆xp1∆tp

(sin(kωtp)− sin(kωtp−1))

ck =1

πωk2

P∑

p1

∆xp2∆tp

(cos(kωtp)− cos(kωtp−1))

dk =1

πωk2

P∑

p1

∆xp2∆tp

(sin(kωtp)− sin(kωtp−1))

T – the period of the closed curve and ω = 2πT.


Outline

Shape Descriptors

Edges


RANSAC

Bounding Box


Shape from Shading

Problem Statement

From a monocular view with a single distant light source of knownincident orientation upon an object with known reflectance map,solve for the normal map.


Outline

Shape Descriptors

Edges


RANSAC

Bounding Box


Shape from Shading

Figure: Shape from shading - Left: Constant intensity, Right: Lambertianshading


Outline

Shape Descriptors

Edges


RANSAC

Bounding Box


Shape from Shading

Surface normal and depth


Outline

Shape Descriptors

Edges


RANSAC

Bounding Box


Shape from Shading

Unique Shape?


Outline

Shape Descriptors

Edges


RANSAC

Bounding Box


Shape from Shading

Surface - Surface Normals

A general surface S :

S =

Xi

Yi

Zi

| i = 1, . . . ,Ns

Note S can be represented by a 2D height function z : R2 → R -Namely

S =

xjyj

z(xj , yj)

| j = 1, . . . ,N

∂S

∂x=

10∂z∂x

∂S

∂y=

01∂z∂y


Outline

Shape Descriptors

Edges


RANSAC

Bounding Box


Shape from Shading

Surface Normals & Shape

Letting p = ∂z∂x

and q = ∂z∂y, then surface normal n

n(x , y) =

10p

×

10q

=

−p

−q

1


Outline

Shape Descriptors

Edges


RANSAC

Bounding Box


Shape from Shading

From Surface Normals To Shape


Outline

Shape Descriptors

Edges


RANSAC

Bounding Box


Shape from Shading

Lambertian Surface - Reflectivity

RecallE = Lρcosθ

Then,cosθ = nTnL

Therefore

R(p, q) =ppL + qqL + 1

√

p2 + q2 + 1√

p2L + q2L + 1


Outline

Shape Descriptors

Edges


RANSAC

Bounding Box


Shape from Shading

Mathematical Formulation

◮ Image Irradiance equation for surface orientation variables p,q:

I (x) = R(p, q)

◮ Underconstrained =⇒ No unique solution

◮ Minimize error in agreement with Image Irradiance Equationover the region of interest

min

∫ ∫

(I (x)− R(p, q))2dx1dx2

◮ Simultaneously ensure regularity

min

∫ ∫

p2x1 + p2x2 + q2x1 + q2x2)+λ(I (x1, x2)−R(p, q))2dx1dx2


Outline

Shape Descriptors

Edges


RANSAC

Bounding Box


Shape from Shading

Continuous Euler Equations

∇2p = −λ(I (x1, x2)− R(p, q))∂R

∂p

∇2q = −λ(I (x1, x2)− R(p, q))∂R

∂q

where

∇2 =∂2

∂x21+

∂2

∂x22


Outline

Shape Descriptors

Edges


RANSAC

Bounding Box


Shape from Shading

Discrete Euler Equations

E (p, q) =∑

i

∑

j

sij + rij

sij =1

4

(

(pi+1,j − pi ,j)2 + (pi ,j+1 − pi ,j)

2

(qi+1,j − qi ,j)2 + (qi ,j+1 − qi ,j)

2)

rij = (Iij − R(pij , qij))2


Outline

Shape Descriptors

Edges


RANSAC

Bounding Box


Shape from Shading

Iterative Procedure

pn+1ij = pnij + λ(Iij − R(pnij , q

nij))

∂R

∂p

qn+1ij = qnij + λ(Iij − R(pnij , q

nij))

∂R

∂q


Outline

Shape Descriptors

Edges


RANSAC

Bounding Box


Shape from Shading

Average Computation

The computation for the average can be computed using thestencil:

1

20

1 4 14 0 41 4 1


Outline

Shape Descriptors

Edges


RANSAC

Bounding Box


Shape from Shading

D. G. Lowe, “Object recognition from local scale-invariantfeatures,” in ICCV, 1999, pp. 1150–1157.

J. Sivic and A. Zisserman, “Efficient visual search of videoscast as text retrieval,” Pattern Analysis and Machine

Intelligence, IEEE Transactions on, vol. 31, no. 4, pp.591–606, April 2009.

A. Torralba, K. P. Murphy, W. T. Freeman, and M. A. Rubin,“Context-based vision system for place and objectrecognition,” Computer Vision, IEEE Int. Conf. on, vol. 1, p.273, 2003.

O. Erkent and H. I. Bozma, “Bubble Space and PlaceRepresentation in Topological Maps,” The Int. J. of Rob. Res.,vol. 32, no. 6, pp. 671 – 688, 2013.


Documents

EE 576 - Shape Descriptorsisl.ee.boun.edu.tr/courses/ee576/lectures/sunum/shaped...Selects N data items at random Estimates parameter Finds how many data items (of M) ﬁt the model