Advanced Computer Vision

Feature-based AlignmentLecturer: Lu Yi & Prof. Fuh

lynn.luyi@gmail.comCSIE NTU

Content• Pre-requisites: mainly about camera model• Alignment Algorithm• RANdom Sample Consensus (RANSAC)• Pose Estimation

Sources• EGGN 512 – Computer Vision course by Prof. Willliam Hoff• http://inside.mines.edu/~whoff/courses/EENG512/• Playlist@Youtube:• http://www.youtube.com/playlist?list=PL4B3F8D4A5CAD8DA3

• Digital Visual Effects course by Prof. Yung-Yu Chuang• http://www.csie.ntu.edu.tw/~cyy/courses/vfx/14spring/news/

• Both are highly recommended

Taught Way• Cover the main parts of textbook chapter 6• Lecture mostly depended on EGGN 512 course videos

Frames of Reference• Image frames are 2D; other are 3D

• The “pose” (position and orientation) of a 3D rigid body has 6 degrees of freedom

Pre-requisites• Image Formation• Geometric Transformations (2D-2D, 3D-3D, 3D-2D)• Camera Model (Camera Matrix = Intrinsic + Extrinsic camera matrix)

• All related contents are in textbook chapter 2: Image formation

Pre-requisites• Review1. Projective Transformation2. Camera Matrix• EGGN 512 – Lecture 5-1 3D-2D Transforms• http://

www.youtube.com/watch?v=DFNjOUMuecU&index=11&list=PL4B3F8D4A5CAD8DA3• EGGN 512 – Lecture 5-2 3D-2D Transforms• http://

www.youtube.com/watch?v=5gesrLgNuQo&list=PL4B3F8D4A5CAD8DA3

Alignment Algorithm• EGGN 512 – Lecture 14-1 Alignment• http://www.youtube.com/watch?v=UcU4814hvR8&list=PL4B3F8D4A5CAD8DA3• EGGN 512 – Lecture 14-2 Alignment• http://

www.youtube.com/watch?v=XxEKMecNZk0&list=PL4B3F8D4A5CAD8DA3• EGGN 512 – Lecture 14-3 Alignment• http://www.youtube.com/watch?v=lSXfv4baMwk&list=PL4B3F8D4A5CAD8DA3• Slides • http://inside.mines.edu/~

whoff/courses/EENG512/lectures/15-AlignmentNonlinear.pdf

Alignment Algorithm• Feature-based methods: only use feature points to estimate

parameters• Features: SIFT, SURF, Harris .etc.

Determine pairwise alignment• p’=Mp, where M is a transformation matrix, p and p’ are feature

matches• It is possible to use more complicated models such as affine or

perspective• For example, assume M is a 2x2 matrix

• Find M with the least square error

yx

mmmm

yx

2221

1211

''

n

i

pMp1

2'

Determine pairwise alignment

• Overdetermined system

yx

mmmm

yx

2221

1211

''

'1221211

'1121111

ymymx

xmymx

'

'

'2

'1

'1

22

21

12

11

22

11

11

0000

0000

00

n

n

nn

nn

yx

xyx

mmmm

yxyx

yxyx

yx

Normal Equation• Given an overdetermined system

• the normal equation is that which minimizes the sum of the square differences between left and right sides

• Why?

bAx

bAAxA TT

Normal Equation 2)( bAxx E

n

m

nmn

m

b

b

x

x

aa

aa

:::

:

...::::::

... 1

1

1

111

nxm, n equations, m variables

Normal Equation

n

m

jjnj

i

m

jjij

m

jjj

n

i

m

jjnj

m

jjij

m

jjj

bxa

bxa

bxa

b

b

b

xa

xa

xa

1

1

11

11

1

1

11

:

:

:

:

:

:

bAx

n

ii

m

jjij bxaE

1

2

1

2)( bAxx

Normal Equation

n

ii

m

jjij bxaE

1

2

1

2)( bAxx

1

0xE

n

iii

n

ij

m

jiji baxaa

11

1 11 22

11 1

2 i

n

ii

m

jjij abxa

)(20 bAAxAx

TT

E bAAxA TT →

Normal Equation 2bAx

Normal Equation

bbxbAxbAAxAx

bbbAxAxbAxAxbAxbAxbAxbAx

bAxbAx

bAx

TTTTTTT

TTTTTT

TTT

TT

T

2

bAAxAx

TT 22 E

Determine pairwise alignment• p’=Mp, where M is a transformation matrix, p and p’ are feature

matches• For translation model, it is easier.

• What if the match is false? Avoid impact of outliers. (Ans: RANSAC)

n

iiiii yymxxmE

1

2'2

2'1

1

0mE

Application: Image Stitching• Stitching = alignment + blending

geometricalregistration

photometricregistration

Application: Image Stitching• Panorama – the whole picture

• Compact Camera FOV = 50 x 35°

Application: Image Stitching• Panorama – the whole picture

• Compact Camera FOV = 50 x 35°• Human FOV = 200 x 135°

Application: Image Stitching• Panorama – the whole picture

• Compact Camera FOV = 50 x 35°• Human FOV = 200 x 135°• Panoramic Mosaic = 360 x 180°

Application: Image Stitching• Example:• http://graphics.stanford.edu/courses/cs178/applets/projection.html

Application: Face Alignment• Face alignment is the key preprocessing step for face recognition

Application: Face Alignment• Examples from LFW dataset:

RANSAC Algorithm• RANSAC = RANdom Sample Consensus• An iterative method to estimate parameters of a mathematical model

from a set of observed data which contains outliers [Wiki]• Compare to robust statistics• Given N data points xi, assume that majority of them are generated from a

model with parameters , try to recover .

• Fischler, M. A., & Bolles, R. C. (1981). Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 24(6), 381-395.

RANSAC AlgorithmRun k times: (1) draw n samples randomly (2) fit parameters with these n samples (3) for each of other N-n points, calculate its distance to the fitted model, count the number of inlier points, cOutput with the largest c

How many times?How big? Smaller is better

How to define?Depends on the problem.

RANSAC Algorithm (How to determine K)

p: probability of real inliersP: probability of success after k trials

knpP )1(1 n samples are all inliers

a failurefailure after k trials

)1log()1log(

npPk

n p k

3 0.5 356 0.6 976 0.5 293

for P=0.99

RANSAC Algorithm (Example: line fitting)

RANSAC Algorithm (Example: line fitting)

n=2

RANSAC Algorithm (Example: line fitting)

RANSAC Algorithm (Example: line fitting)

RANSAC Algorithm (Example: line fitting)

c=3

RANSAC Algorithm (Example: line fitting)

c=3

RANSAC Algorithm (Example: line fitting)

c=15

RANSAC Algorithm• EGGN 512 – Lecture 27-1 RANSAC• http://www.youtube.com/watch?v=NKxXGsZdDp8&list=PL4B3F8D4A5

CAD8DA3&index=54

RANSAC Algorithm (Example: Homography)

RANSAC Algorithm (Example: Homography)

RANSAC Algorithm (Example: Homography)

Pose Estimation• EGGN 512 – Lecture 16-1 Pose Estimation• http://

www.youtube.com/watch?v=kq3c6QpcAGc&list=PL4B3F8D4A5CAD8DA3

• EGGN 512 – Lecture 17-1 Pose from Lines• http://

www.youtube.com/watch?v=D_4eUoqgWdc&list=PL4B3F8D4A5CAD8DA3

Pose Estimation• EGGN 512 – Lecture 18-1 SVD• http://

www.youtube.com/watch?v=C852P1JrHXI&list=PL4B3F8D4A5CAD8DA3

• EGGN 512 – Lecture 18-2 SVD• http://

www.youtube.com/watch?v=aIkzK4CdYes&list=PL4B3F8D4A5CAD8DA3

Pose Estimation• EGGN 512 – Lecture 19-1 Linear Pose Estimation• http://

www.youtube.com/watch?v=HojSSrxsB4Q&list=PL4B3F8D4A5CAD8DA3• EGGN 512 – Lecture 19-2 Linear Pose Estimation• http://

www.youtube.com/watch?v=ik8dFybnyPY&list=PL4B3F8D4A5CAD8DA3• EGGN 512 – Lecture 19-3 Linear Pose Estimation• http://

www.youtube.com/watch?v=OS5b-3Xfn1M&list=PL4B3F8D4A5CAD8DA3

END