875: Recent Advances in Geometric Computer Vision & Recognition

Jan-Michael FrahmFall 2011

2

Introductions

3

Class• Recent methods in computer vision• Broadness of the class depends on YOU!• Initial papers in class are the best papers of

CVPR(2010,2011), ICCV(2011), ECCV(2010)

4

Grade Requirements• Presentation of 3 papers in class

30 min talk, 10 min questions

• Papers for selection must come from: top journals: IJCV, PAMI, CVIU, IVCJ top conferences: CVPR (2010,2011), ICCV (2011),

ECCV (2010), MICCAI (2010, 2011) approval for all other venues is needed

• Final project evaluation, extension of a recent method from

the above

5

Schedule• Aug. 29th, Introduction• Aug. 31st, Geometric computer vision, first paper selection• Sept. 5th, Labor day• Sept. 7th, Geometric computer vision, Robust estimation• Sept. 12th, Optimization• Sept. 19th, Classification• Sept. 21st, 1. round of presentations starts• Oct 17th, 2. round of presentations starts• Oct 31st, definition of final projects due• Nov 7th, 3. round of presentations starts• Dec 5th and 7th, final project presentation

6

How to give a great presentation

• Structure of the talk: Motivation (motivate and explain the

problem) Overview Related work (short concise discussion) Approach Experiments Conclusion and future work

7

How to give a great presentation

• Use large enough fonts 5-6 one line bullet items on a slide

max• Keep it simple• No complex formulas in your talk• Bad Powerpoint slides• How to for presentations

8

How to give a great presentation

• Abstract the material of the talk provide understanding beyond

details• Use pictures to illustrate

find pictures on the internet create a graphic (in ppt, graph tool) animate complex pictures

9

How to give a good presentation

• Avoid bad color schemes no red on blue looks awful

• Avoid using laser pointer (especially if you are nervous)

• Add pointing elements in your presentation

• Practice to stay within your time! • Don’t rush through the talk!

Projective and homogeneous points• Given: Plane in R2 embedded in P2 at coordinates w=1

viewing ray g intersects plane at v (homogeneous coordinates) all points on ray g project onto the same homogeneous point v projection of g onto is defined by scaling v=g/l = g/w

w=1

R3 ( )1

x xg v y y

w

1

1 1

xwxyv y g

w w

O

(R2)

w

x

v

y

Affine and projective transformations Affine transformation leaves infinite points at infinity

11 12 13

21 22 23

31 32 33

0 0 0 0 1 0

x

y

z

X a a a t XY a a a t YZ a a a t Z

'affineM T M

Projective transformations move infinite points into finite affine space

'projectiveM T M

11 12 13

21 22 23

31 32 33

41 42 43 441 0

xp

yp

zp

a a a tx X Xa a a ty Y Ya a a tz Z Zw w w ww

Example: Parallel lines intersect at the horizon (line of infinite points). We can see this intersection due to perspective projection!

Homogeneous coordinates

0 cbyax Ta,b,c0,0)()( kkcykbxka TT a,b,cka,b,c ~

Homogeneous representation of lines

equivalence class of vectors, any vector is representativeSet of all equivalence classes in R3(0,0,0)T forms P2

Homogeneous representation of points

0 cbyax Ta,b,cl Tyx,x on if and only if

0l 11 x,y,a,b,cx,y, T 0,1,,~1,, kyxkyx TT

The point x lies on the line l if and only if xTl=lTx=0

Homogeneous coordinatesInhomogeneous coordinates Tyx,

T321 ,, xxx but only 2DOF

Ideal points and the line at infinity

T0,,l'l ab

Intersections of parallel lines

TT and ',,l' ,,l cbacba

Example

1x 2x

Ideal points T0,, 21 xxLine at infinity T1,0,0l

l22 RP

normal direction ba,

Note that in P2 there is no distinction between ideal points and others

Pinhole Camera Model

Camera obscura

(Frankreich, 1830)

object

optical axis

image plane

principal point (u,v)

Pinhole Camera Model

pmfocal length, aspect

ratio (f, af)

Skew s

01 1

0 0 1

p p

f s um m

m K af v

aperture

Pinhole Camera Model

Selbstkalibrierung bestimmt die intrinsischen Kameraparameter

01 1

0 0 1

p p

f s um m

m K af v

Introduction to Computer Vision for Robotics

Projective Transformation

Projective Transformation maps M onto Mp in P3 space

0

01

1 0 0 00 1 0 00 0 1 00 0 01 1

p

pp p

z

x X Xy Y Y

M T MZ Z Z

W

X

YO

M

pM

Projective Transformation linearizes projection

0

projective scale factorZZ

Introduction to Computer Vision for Robotics

Projection in general pose

Rotation [R]

Projection center C M

World coordinates

Projection:m

p

0 1cam T

R CT

P

pm PM

1

0 1

T T

scene cam T

R R CT T

Introduction to Computer Vision for Robotics

Projection matrix P Camera projection matrix P combines:

inverse affine transformation Tcam-1 from general pose to origin

Perspective projection P0 to image plane at Z0 =1 affine mapping K from image to sensor coordinates

0, = T Tscenem PM P KPT K R R C

0

1 0 0 0projection: 0 1 0 0 0

0 0 1 0P I

scene pose transformation: 0 1

T T

scene T

R R CT

sensor calibration: 00 0 1

x x

y y

f s cK f c

Homography

X

Y0

M

im

jm

Homography plane to plane warping purely rotating camera

m

Self-calibration for Rotating CamerasAgapito et al.

1 1, , ,j i j i i j i jR K H K

• Rotation invariant formulation

2 1, , , , ,

T T T Tj i j i j i i j i j j j i iR R K H K K H K I

2, , ,

T T Ti i j i j i j j j iK K H K K H

Projection of the dual absolute conic into image i

Projection of the dual absolute conic into image j

, ,T

j i j iR R I

1, , ,j i j i i j i jH K R K

calibration through Choleski decomposition

Ti iK K

Removing projective distortion

333231

131211

3

1

'''

hyhxhhyhxh

xxx

333231

232221

3

2

'''

hyhxhhyhxh

xxy

131211333231' hyhxhhyhxhx 232221333231' hyhxhhyhxhy

select four points in a plane with know coordinates

(linear in hij)

(2 constraints/point, 8DOF 4 points needed)

Remark: no calibration at all necessary

Freely Moving Camera

XY

Cj

jm

M

Z

Ci

Epipolar Linie:

im

ie

m

iml

im

Computable from image correspondences

Example: motion parallel with image plane

Example: forward motion

e

e’

Introduction to Computer Vision for Robotics

The Essential Matrix E• F is the most general constraint on an image pair. If the camera

calibration matrix K is known, then more constraints are available

• Essential Matrix E

E holds the relative orientation of a calibrated camera pair. It has 5 degrees of freedom: 3 from rotation matrix Rik, 2 from direction of translation e, the epipole.

010101~~~~ mFKKmmKFmKFmm

E

TTTT

00

0][ with ][

xy

xz

yz

xx

eeeeee

eReE

Estimation of P from E• From E we can obtain a camera projection matrix pair:

E=Udiag(0,0,1)VT

• P0=[I3x3 | 03x1] and there are four choices for P1:

P1=[UWVT | +u3] or P1=[UWVT | -u3] or P1=[UWTVT | +u3] or P1=[UWTVT | -u3]

100001010

with W

four possible configurations:

only one with 3D point in front of both cameras

Kruppa Equations

, ,[ ]j i j i x i j iF K e K R

, , , ,[ ] [ ]T T

j i j j i j i x i j i i x i j iF K F K e K R e K R

, , [ ] [ ]T T T Tj i j i i x i xF K K F e K K e

1, ,[ ]j i i x i j i jF e K R K

Kruppa-equation (Faugeras et al.`92)

for constant camera calibration

Dual absolute conic limited to epipolar-geometrie