Vision-based Registration for AR

Presented by Diem Vu

Nov 20, 2003

Markerless Tracking using Planar Structure in the Scene. G. Simon, A.W. Fitzgibbon and A. Zisserman, 2000.

Calibration-Free Augmented Reality. K.N Kutulakos and J.R. Vallino, 1998.

Planar-surface tracking.

Camera position can be recovered from planar homography.

Planar structure is common in almost all scenarios.

y

x

z

Hw

World to image homography

jiH

Image to image homography

World to image homography

Consider our tracking plane is the plane Z=0

y

x

z

Hw

1

H

1w Y

X

y

x

Projection matrix

trrr 321KP

trrrr 2121 KP

Projection matrix

1

y

x

1

0

Y

X

y

x

z

P

1

0K

12

Y

X

y

x

trrr 31

trrrr 2121 KP

Projection matrix

1

y

x

1

0

Y

X

y

x

z

P

1

K

1

Y

X

y

x

trr 21

If K and Hw are known, then r1, r2 and t can be recovered, thus P.

Question: How to compute Hw?Direct.Indirect.

trr 21KHw

Direct measurement of Hw

Select 4 points {xk} on a rectangle in the scene.

Compute H which maps the unit square to {xk}.

(0,0)

(0,1) (1,1)

((1,0))

Direct measurement of Hw

Select 4 points {xk} on a rectangle in the scene.

Compute H which maps the unit square to {xk}.

s,1)(1,diagH H w

trr 21 s HK -1

s,1)(1,diagK H trr 21

Compute Hw=Hdiag(1,1/s,1)

(0,0)

(0,s) (1,s)

((1,0))

Indirect measurement of Hw

iwH

jiH

? H jw

y

x

z

Indirect measurement of Hw

iwH

jiH

iw

ji

jw HHH

y

x

z

Algorithm summary

Compute (direct measure).For each frame i, compute frame to frame

homography (RANSAC)Compute by:

0wH

1-iw

i1-i

iw HHH

i1-iH

iwH

Other …

Using only 2 points in direct method ??Matching the frame i with frame 0 in order

to reduce error.Estimate intrinsic parameters K Hand-off mechanism.

Possible problems?

Homography is only up-to-scale?Plain surface (no texture) or moving

objects in the foreground ?Depth order, occlusion ?Speed ?

Affine virtual object representation

Represent virtual objects so that their projection can be computed as a linear combination of the projection of the fiducial points.

Project a point from its affine coordinates

Compute affine coordinates from projection along two viewing

direction

Algorithm

Setup the affine basis

Algorithm

Setup the affine basis Locate the object in 2 frames.

Algorithm

Setup the affine basis Locate the object in 2 frames. Compute the affine coordinates

for each point.

Algorithm

Setup the affine basis Locate the object in 2 frames. Compute the affine coordinates

for each point. Compute projection of the object

and render the object in each frame.

Camera viewing direction

and are the first and second row of 2x3.

The camera viewing direction expressed in the coordinate frame of the affine basis points: =

Depth order

w is the z-value of point p (x,y,z).

Advantages

No need any metric information.Able to use with the existing hardware to

accelerate graphics operations.Can be used to improve tracking.

Limitation

Affine constraints.Lost of metric information.