Paper Reading

Dalong Du

Nov.27, 2009

Papers

• Leon Gu and Takeo Kanade. A Generative Shape Regularization Model for Robust Face Alignment. ECCV08.

• Yan Li, Leon Gu, Takeo Kanade. A Robust Shape Model for Multi-view Car Alignment. CVPR09.

A Generative Shape Regularization Model for Robust Face Alignment

Leon Gu and Takeo Kanade

Outline

• Author Introduction.

• Problem Introduction.

• How to do?

• Discussion.

Outline

• Author Introduction.

• Problem Introduction.

• How to do?

• Discussion.

Author Introduction (1/3)

Leon Gu Takeo Kanade(金出武雄 )

Author Introduction (2/3)

• Leon Gu– Ph.D. candidate in the Computer Science

Department at Carnegie Mellon University, advised by Professor Takeo Kanade.

– His main research interest is in developing robust and efficient algorithms for object recognition. A common thread has been the focus on reasoning the shape of visual objects from noisy, real-world images, where the uncertainties over image appearance and imaging conditions are prevalent.

Author Introduction (3/3)

• Takeo Kanade– Director of the Robotics Institute of Carnegie

Mellon University– Wisdom：像外行一样思考，像专家一样实践

Outline

• Author Introduction.

• Problem Introduction.

• How to do?

• Discussion.

Problem Introduction (1/12)

• Face Q consists of N landmark points:– – The geometry information of Q decouples into

two parts:• A canonical shape S

– b– e.g. Or other

linear or nonlinear methods

• A transformation– θ– e.g. similarity s, R, t Or

Affine or others.

x u b b

θ

Problem Introduction (2/12)

• Probabilistic Formulation– Generic alignment problem

•

Where– Pose space Θ is free– Shape space is constrained

• A solution maximizes the posterior

• A chicken and egg problem– A best solution A max posterior

Problem Introduction (3/12)

• In the Eyes of Computer– On the basis of such “noisy observation”,

how can make the best hypothesis (b, θ) ?

Reflectance,Occlusion,Image blur,…..

Noisy feature map

Deformation

Transformation

Image Likelihood

Problem Introduction (4/12)

• A Generative hierarchical model– Deformation

• The magnitude of deformation is controlled by b.

• The canonical shape S is generated from b

through , a process that could be linear or nonlinear.

– Transformation • The transform could be similarity/affine.

– Image Likelihood• Varies with the type of image local feature

– Profile, local image patch, Haar-like feature… …

Deformation

Transformation

Image Likelihood

Problem Introduction (5/12)

• Baseline Model– Linear Deformation

•

Where– Shape prior , Λ is diagonal.– Isotropic shape noise (Probabilistic PCA)– The average residual variance outside of the subspace

, where N is the number of landmark points, M is the subspace dimension.

• {Φ, μ, σ} are learned from training samples.

22

1

1

2

N

mm MN M

Deformation

Transformation

Image Likelihood

Problem Introduction (6/12)

• Baseline Model– Similarity Transform

•

Where– θ={s, R, t} are scale, Rotation, translation coefficients

respectively.– Diagonal observation noise

» measures the noise level of the observation of n-th landmark point.

» » Σ is also learned from training samples.

n

( ( ), )n Dist T S Y

Deformation

Transformation

Image Likelihood

Problem Introduction (7/12)

• Baseline Model– Observed shape Y is generated from

feature point detector.–

• EM– Q-function:– E-step: compute the statistics that are required to

evaluate Q-function.– M-step: maximize Q-function to find the updated shape

and pose.

( , , | ) ( | , ) ( | )p b S Y p Y S p S b Bayes

log ( , , | ) log ( | , ) log ( | )S S S

p b S Y p Y S p S b

Problem Introduction (8/12)

• Alignment algorithm

Problem Introduction (9/12)

• Problems?– Linear deformation model

• Cannot handle faces of rare shapes (babies, etc)• Cannot handle extreme expressions

– Single candidate position for each feature point• Best position may be the one with second strongest

response– This paper extends the generative framework to

handle• Large face shape deformation including extreme

expressions• Multiple candidate positions for each feature point• Identify outliers, like occluded feature points.

Problem Introduction (10/12)

• Handling Extreme Expressions

Problem Introduction (11/12)

• Handling Large Occlusion

Problem Introduction (12/12)

• Handling Real World Images

Outline

• Author Introduction.

• Problem Introduction.

• How to do?

• Discussion.

How to do? (1/12)

• Face Q consists of N landmark points:– – The geometry information of Q decouples into

two parts:• A canonical shape S• A similarity transformation

– Map S from a common reference

frame to the coordinate plane of

the image I

b

θ

How to do? (2/12)

• Make a mixture of constrained Gaussian– Multiple subspace–

How to do? (3/12)

• Allow generate multiple candidate– For n-th landmark

• K candidate positions • denote the whole set of N × K candidates• Set a binary N × K matrix h to specify the “true”

candidate

11 12 1 12

21 22 2 21

1 2 1

... 0 1 ... 0

... 1 0 ... 0( )

... ... ... ... ...... ... ... ...

... 0 0 ... 1

K

K

N N NK NKN KN K N

Q Q Q Q

Q Q Q QQ h Q h

Q Q Q Q

e.g.

How to do? (4/12)

• A new generative hierarchical model

Deformation

Transformation

Image Likelihood

S

1b Lblb ......

nk N KQ Q

z

h

I

Deformation

Transformation

Image Likelihood

( )Q h

How to do? (5/12)

• Deformation– Define prior distribution over the shape S as a

mixture of Gaussian

– Introduce a multinomial distribution z

– Model parameters learned from training samples

, , ,l l l l

0

1

...

0

z

e.g.

How to do? (6/12)

• Similarity Transform

Where– θ={s, R, t} are scale, Rotation, translation

coefficients respectively.– Diagonal observation noise

• measures the noise level of the observation of n-th landmark point.

• • Σ is also learned from training samples and can

updated on fitting phase.– So

n

( ( ), )n Dist T S Y

( )Q h sRS t

How to do? (7/12)

• Image Likelihood– The image likelihood of seeing a landmark

atone particular position Qnk is measured by •

– is generated by feature detector. nk

How to do? (8/12)

• Goal:– Solve b and θ on the basis of the candidate point

set Q.

• MAP problem which can be solved by EM– Posterior with latent variables S, h, z

– Take the expectation of the log over the posterior of the latent variables S, h, z

• Q function:

How to do? (9/12)

• Alignment Algorithm

How to do? (10/12)

• Update Canonical Shape

How to do? (11/12)

• Update Shape Parameters

Shrink by:

How to do? (12/12)

• Identifying Outliers:– Use observation noise model

– Observation noises are unpredictable• Update online

– Change it according to the fitting error between the model prediction and the averaged candidate position

• Define weights to update Canonical Shape– A smaller leads a larger weight to the canonical

shape and less to the observed candidate.

( ( ), )n Dist T S Y

( )Q h sRS t

n

Outline

• Author Introduction.

• Problem Introduction.

• How to do?

• Discussion.

Discussion (1/6)

• Evaluation

Discussion (2/6)

• Handling Extreme Expressions– Number of mixture components is L = 3.

Discussion (3/6)

• Handling Large Occlusion

Discussion (4/6)

• Handling Real World Images

Discussion (5/6)

• Similarity Transform

Where– Diagonal observation noise

• measures the noise level of the observation of n-th landmark point.

• The independence assumption to each landmark is not reasonable.– Markov Network– …

n

( )Q h sRS t

Discussion (6/6)

• The regularization step does not consider the image information anymore

A Robust Shape Model for Multi-view Car Alignment

Yan Li, Leon Gu and Takeo Kanade

Outline

• Problem Introduction.

• How to do?

• Discussion.

Outline

• Problem Introduction.

• How to do?

• Discussion.

Problem Introduction

• Previous shape alignment model– A hypothesis of Gaussian observation noise.– Use all the observed data to fit a regularized

shape.

• This Gaussian assumption is vulnerable to gross feature detection error.

Partial occlusions and spurious background features

Outline

• Problem Introduction.

• How to do?

• Discussion.

How to do? (1/3)

• A hypothesis-and-test approach.– Hypothesis: Bayesian Partial Shape Inference

(BPSI) algorithm•

– Test: The hypotheses are then evaluated to find the one that minimizes the shape prediction error.

1 2{ , ,..., }, { , } { , }N n n nSubsets Q Q Q Q b

How to do? (2/3)

• The observed data– – Random sample from Y

• used to generate hypothesis—shape b and pose θ(s, R, t).

• used to test hypothesis.

• Bayesian Partial Shape Inference (BPSI) algorithm– A MAP problem:– A typical missing data problem can be solved by

EM.

1 2{ , ,..., }, { , }N n n nY Q Q Q Q

pY

hY

How to do? (3/3)

Generate Hypothesis

Test Hypothesis

is the residual between the i-th Corresponding point ofi

and p hY Y

Discussion

Discussion

Thank you!