Mysteries of Parameterizing Camera

Motion - Part 2Instructor - Simon Lucey

16-623 - Advanced Computer Vision Apps

Today

• Gauss-Newton and SO(3)

• Alternative Representations of SO(3)

• Exponential Maps

• SL(3) for Homographies

Example - Extrinsic Estimation

argmin✓

NX

n=1

||xn � ⇡(wn;✓)||22

x1

x2

x3

x4

✓ = [⌦, ⌧ ] “extrinsics”⇡(·) ! R3 : R2 “projection function”

Example - Extrinsic Estimation

argminy

||x� F(y)||22

Example - Structure from Motion

argminy

||x� F(y)||22

Reminder: Non-Linear Least Squares

• Many problems in vision can be expressed as solving a non-linear least-squares objective.

y

||x�F(y

)||2 2

Reminder: Gauss-Newton• Attempts to solve a non-linear least-squares problem by

approximating it at as a sequence of linear least-squares problems.

7

• If initialized properly, Gauss-Newton has quadratic convergence properties unlike steepest descent which is at most linear.

||x�F(y

)||2 2

y

Reminder: Gauss-Newton Algorithm

• Gauss-Newton algorithm common strategy for optimizing non-linear least-squares problems.

8

s.t. F : RN ! RM

Step 1:

Step 2:

keep applying steps until converges.

“Carl Friedrich Gauss”

“Isaac Newton”

argminy

||x� F(y)||22

argmin�y

||x� F(y)� @F(y)

@yT�y||22

y ! y +�y

�y

“Problem”

Reminder: Non-Convex Set

9

What happens if y belongs to a non-convex set?

Reminder: SO(3) is not a Convex Domain

• As is a rotation matrix it is constrained by the following,

• We refer to these matrices as belonging to the Special Orthogonal Group - SO(3).

⌦

⌦T⌦ = I

det(⌦) = 1

⌦1,⌦2 2 SO(3)

�⌦1 + (1� �)⌦2

s.t. 0 � 12/ SO(3), 8�

Today

• Gauss-Newton and SO(3)

• Alternative Representations of SO(3)

• Exponential Maps

• SL(3) for Homographies

Non-Uniqueness of Euler Angles

• We can define a rotation matrix through Euler angles.

12

• Euler angles do form a convex set.

• However, the representation is not unique as it is possible,

⌦(✓u, ✓v, ✓w) = ⌦(✓u)⌦(✓v)⌦(✓w)

⌦(✓u, ✓v, ✓w) = ⌦(✓0u, ✓0v, ✓

0w)

✓u 6= ✓0u ✓v 6= ✓0v ✓w 6= ✓0w

�⇡ ✓ ⇡

Example - Gimbal Lock

13

• Extension of complex numbers they take the form,

• Unlike Euler angles, unit quaternions give a global parameterization of SO(3), at the cost of using four numbers instead of three to represent a rotation.

Quaternions

14

where i, j and k are fundamental quaternion units.

“William Rowan Hamilton”

q0 + q1 · i+ q2 · j + q3 · k

q =

2

664

q0q1q2q3

3

775 s.t. ||q||22 = 1

Quaternions - Problem

15

q0

q1

||q||22 = 1

q =

q0q1

�

Does q form a convex set?

Today

• Gauss-Newton and SO(3)

• Alternative Representations of SO(3)

• Exponential Maps

• SL(3) for Homographies

Right-hand rule

• Given three orthonormal 3D vectors (x,y, z)

z = x⇥ y

• Then,

Uniquely Defining Rotation

18

n

✓

||n||2 = 1 �⇡ ✓ ⇡

Exponential Map

19

n

✓

w(0)

w(✓)

w(✓) = ⌦(n, ✓)w(0)

Exponential Map

20

w(0)

w(✓)

n

@w(0)

@✓= [n]⇥w(0)

@w(0)

@✓

Exponential Map

21

nw(�✓)

w(�✓) ⇡ (I+�✓ · [n]⇥)w(0)

Exponential Map

22

n

w(✓) = (I+ ✓ · [n]⇥ +✓

2!· [n]2⇥ +

✓

3!· [n]3⇥ + . . .)w(0)

w(✓)

w(0)

Exponential Map

23

n

w(✓)

w(✓) = exp(✓ · [n]⇥)w(0)

Exponential Map

24

• To more compactly represent a 3D rotation matrix,

• Would this work in MATLAB???

>> skew = @(n) [0,-n(3),n(2);… n(3),0,n(1);… -n(2),n(1),0]; >> R = exp(theta*skew(n));

⌦ = exp(✓[n]⇥)

>> R = expm(theta*skew(n));

Exponential Map = Convex Set

⌦(!) = exp([!]⇥)

! = ✓ · nwhere,

||n||2 = 1

�⇡ ✓ ⇡therefore,

! 2 R3

s.t. ||!||2 ⇡

!

Rodrigues Formula

• Rodrigues formula parametrizes a 3D rotation matrix uniquely in terms of the axis and angle.

26

n ✓

a⇥ b = [a]⇥b =

2

40 �a3 a2a3 0 �a1�a2 a1 0

3

5where,

R.M. Murray, Z. Li, and S.S. Sastry. “A Mathematical Introduction to Robotic Manipulation”. CRC Press, 1994.

Benjamin Olinde Rodrigues (6 October 1795 – 17 December 1851)

⌦ = I+ sin(✓)[n]⇥ + [1� cos(✓)][n]2⇥

[a]2⇥ = aaT � ||a||22I[a]3⇥ = �||a||22[a]⇥

Short Angle Approximation

• For many optimization problems in vision involving rotation matrices people try to take advantage of the short angle approximation.

27

sin(✓) ⇡ ✓cos(✓) ⇡ 1� ✓2

• In MATLAB type,

>> x = linspace(-1,1);

>> plot(x,sin(x),x,x,x,cos(x),x,1-x.*x);

Short Angle Approximation?

28

f(✓) = sin(✓)

f(✓) = cos(✓)

f(✓) = ✓

f(✓) = 1� ✓2

✓

f(✓)

How could this be used to simplify linearization of ?⌦

Linearizing

29

⌦(! +�!) ⇡ ⌦(!) +@⌦(!)

@!x

�!x

+@⌦(!)

@!y

�!y

+@⌦(!)

@!z

�!z

⌦

• Naively we would linearize,

• Using the short angle approximation why is it better to?

⌦(�!) ⇡ I+ [�!]⇥Hint:

⌦(!)⌦(0+�!) ⇡ ⌦(!)

⇢@⌦(0)

@!x

�!x

+@⌦(0)

@!y

�!y

+@⌦(0)

@!z

�!z

�

! = [!x

,!y

,!z

]Twhere

||�!||22when is small!!!!

Reminder: Gauss-Newton Algorithm

• Gauss-Newton algorithm common strategy for optimizing non-linear least-squares problems.

30

s.t. F : RN ! RM

Step 1:

Step 2:

keep applying steps until converges.

“Carl Friedrich Gauss”

“Isaac Newton”

argminy

||x� F(y)||22

argmin�y

||x� F(y)� @F(y)

@yT�y||22

y ! y +�y

�y

“Is the update additive?”

Today

• Gauss-Newton and SO(3)

• Alternative Representations of SO(3)

• Exponential Maps

• SL(3) for Homographies

Reminder: Homographies

�1 �2

�3 = �2��11

• Scale ambiguity is controlled by

Homographies are not Convex

“Scale Ambiguous”

det(�) = +1

Constraint is not convex!!!

SL(3) Matrices

• Turns out any 3x3 matrix that has the constraint,

det(�) = +1

• called Special Linear Group of dimension 3 - SL(3). • Similar trick can be employed such that,

�(�) = exp

8X

k=1

�kAk

!

� 2 R8

SL(3) Matrices

SL(3) on R2 on the left (i.e. w<H><p> ∈ R2). The mapw<H> : R2 → R2 defines a coordinate transformation (awarping) such that q = w<H><p>. Let I be the identityelement of the transformation group :

• w<I><p> is the identity map. i.e. ∀p ∈ R2:

w<I><p> = p (1)

• the composition of two actions corresponds to the actionof the composition, i.e ∀p ∈ R2, ∀H1, H2 ∈ SL(3):

w<H1><w<H2><p>> = w<H1H2><p> (2)

B. The Lie algebra of SL(3) and the exponential mapLet Ai, with i ∈ {1, 2, ..., 8}, be a basis of the Lie algebra

sl(3) (i.e. the dimension of sl(3) is 8). Any matrix A ∈ sl(3)can be written as a linear combination of the matrices Ai:

A(x) =8∑

i=1

xiAi (3)

where x = (x1, x2, ..., x8) is a (8×1) vector and xi is thei-th element of the base field. The Ai matrices are defined as:

A1 =

⎡

⎣001000000

⎤

⎦,A3 =

⎡

⎣010000000

⎤

⎦,A5 =

⎡

⎣1 0 00−100 0 0

⎤

⎦,A7 =

⎡

⎣000000100

⎤

⎦

A2 =

⎡

⎣000001000

⎤

⎦,A4 =

⎡

⎣000100000

⎤

⎦,A6 =

⎡

⎣0 0 00−100 0 1

⎤

⎦,A8 =

⎡

⎣000000010

⎤

⎦

(4)

The exponential map links the Lie algebra to the Lie Group:exp : sl(3) → SL(3). There exist an open cube v about 0 insl(3) and an open neighborhood U of the identity matrix I

in SL(3) such that exp : v → U is smooth and one-to-oneonto, with a smooth inverse. The neighborhood U of I is verylarge. Consequently, the homography matrix H can be locallyparameterized as:

H(x) = exp

(8∑

i=1

xiAi

)

III. GENERALIZED HOMOGRAPHY-BASED TRACKINGA. Unified projection modelFor sake of completeness, we present here a slightly modi-fied version of the projection model of Geyer [3] and Barreto[4] (Fig. 1). The projection of 3D points can be done in thefollowing steps (the values for ξ and γ are detailed Table I):1) world points in the mirror frame are projected onto theunit sphere, (X )Fm

→ (X s)Fm= X

∥X∥ = (Xs, Ys, Zs)2) the points are then changed to a new reference framecentered in Cp = (0, 0, ξ), (X s)Fm

→(X s)Fp=

(Xs, Ys, Zs − ξ)3) we then project the point onto the normalized plane,

m = ( Xs

Zs−ξ , Ys

Zs−ξ , 1) = !(X s)4) the final projection involves a generalized camera pro-jection matrix K (with γ the generalized focal length,

X

Xs

z⃗m

x⃗s

z⃗s

y⃗m

ξ

1

m

p

πm

Fm

Fp

Cmx⃗m

K

y⃗s

Cp

πp

Fig. 1. Unified image formation

(u0, v0) the principal point, s the skew and r the aspectratio)

p = Km =

⎡

⎣γ γs u0

0 γr v0

0 0 1

⎤

⎦m = k(m)

The function ! is bijective and

!−1(m) =

⎡

⎢⎢⎢⎣

−ξ−√

1+(1−ξ2)(x2+y2)

x2+y2+1 x−ξ−√

1+(1−ξ2)(x2+y2)

x2+y2+1 y−ξ−√

1+(1−ξ2)(x2+y2)

x2+y2+1 + ξ

⎤

⎥⎥⎥⎦(5)

We will call lifting the calculation of the point X s corre-sponding to a given point m (or p according to the context).We may note that in the perspective case, there is no mirrorand only points with Z > 0 are accepted (we thus fall backto the standard projection model with an extra normalizationto the sphere).

TABLE IUNIFIED MODEL PARAMETERS

ξ γParabola 1 −2pf

Hyperbola df√

d2+4p2

−2pf√

d2+4p2

Ellipse df√

d2+4p2

2pf√

d2+4p2

Planar 0 -fPerspective 0 f

d: distance between focal points4p: latus rectum

B. Minimization problemLet I∗ be the reference image. We will call reference tem-plate, a region of size q (rows × columns) of I∗ correspondingto the projection of a planar region of the scene.Two planar points are related by a homography H by

X = HX∗ so the projection of points, belonging to a planar

�(�) = exp

8X

k=1

�kAk

!

C. Mei, S. Benhimane, E. Malis and P. Rives, “Homography-based Tracking for Central Catadioptric Cameras”

Lie Algebra

• Exponential maps on the SO(3) and SL(3) groups are related to the much broader topic of Lie Algebra.

• More details on this topic can be found at in Murray et al. 1994.

• Another group of particular interest in this group will be the SE(3) group.

“Sophus Lie”

T =

⌦ ⌧0T 1

�

More to read…

• R.M. Murray, Z. Li, and S.S. Sastry. “A Mathematical Introduction to Robotic Manipulation”. CRC Press, 1994. (Excellent!!!)

A Mathematical Introduction to

Robotic Manipulation

Richard M. MurrayCalifornia Institute of Technology

Zexiang LiHong Kong University of Science and Technology

S. Shankar Sastry

University of California, Berkeley

c⃝1994, CRC PressAll rights reserved

This electronic edition is available fromhttp://www.cds.caltech.edu/∼murray/mlswiki.

Hardcover editions may be purchased from CRC Press,http://www.crcpress.com/product/isbn/9780849379819.

This manuscript is for personal use only and may not be reproduced, inwhole or in part, without written consent from the publisher.