KLT tracker & triangulationClass 6

Read Shi and Tomasi’s paper on good features to trackhttp://www.unc.edu/courses/2004fall/comp/290/089/papers/shi-tomasi-good-features-cvpr1994.pdf

Optional: Lucas-Kanade 20 Years On http://www.ri.cmu.edu/projects/project_515.html

Feature matching vs. tracking

Extract features independently and then match by comparing descriptors

Extract features in first images and then try to find same feature back in next view

What is a good feature?

Image-to-image correspondences are key to passive triangulation-based 3D reconstruction

Feature point extraction• Approximate SSD for small displacement Δ

• Find points for which the following is maximum

maximize smallest eigenvalue of M

SIFT features

• Scale-space DoG maxima

• Verify minimum contrast and “cornerness”• Orientation from dominant gradient

• Descriptor based on gradient distributions0 2

0 2

Feature tracking

• Identify features and track them over video• Small difference between frames• potential large difference overall

• Standard approach: KLT (Kanade-Lukas-Tomasi)

• Brightness constancy assumption

Intermezzo: optical flow

(small motion)

• 1D example

possibility for iterative refinement

• Brightness constancy assumption

Intermezzo: optical flow

(small motion)

• 2D example

(2 unknowns)

(1 constraint)?

isophote I(t)=Iisophote I(t+1)=I

the “aperture” problem

Intermezzo: optical flow• How to deal with aperture problem?

Assume neighbors have same displacement

(3 constraints if color gradients are different)

Lucas-Kanade

Assume neighbors have same displacement

least-squares:

• Compute translation assuming it is small

Alternative derivation

differentiate:

Affine is also possible, but a bit harder (6x6 in stead of 2x2)

Revisiting the small motion assumption

• Is this motion small enough?• Probably not—it’s much larger than one pixel

(2nd order terms dominate)• How might we solve this problem?

* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003

Reduce the resolution!

* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003

image It-1 image I

Gaussian pyramid of image It-1 Gaussian pyramid of image I

image Iimage It-1u=10 pixels

u=5 pixels

u=2.5 pixels

u=1.25 pixels

Coarse-to-fine optical flow estimation

slides fromBradsky and Thrun

image Iimage J

Gaussian pyramid of image It-1 Gaussian pyramid of image I

image Iimage It-1

Coarse-to-fine optical flow estimation

run iterative L-K

run iterative L-K

warp & upsample

.

.

.

slides fromBradsky and Thrun

Good feature to track

• Tracking

• Use same window in feature selection as for tracking itself

maximize minimal eigenvalue of MStrategy:

• Look for strong well distributed features, typically few hundreds• initialize and then track, renew feature when too many are lost

Example

Simple displacement is sufficient between consecutive frames, but not to compare to reference template

Example

Synthetic example

Good features to keep tracking

Perform affine alignment between first and last frameStop tracking features with too large errors

Live demo

• OpenCV (try it out!)

LKdemo

Triangulation

C1m1

L1

m2

L2

M

C2

Triangulation

- calibration

- correspondences

Triangulation• Backprojection

• Triangulation

Iterative least-squares

• Maximum Likelihood Triangulation

Backprojection

• Represent point as intersection of row and column

Useful presentation for deriving and understanding multiple view geometry(notice 3D planes are linear in 2D point coordinates)

• Condition for solution?

Next class: epipolar geometry