Embed Size (px)
Citation preview
Mo#on Correspondence
photoluver1@flickr
Op#cal Flow
Forward mo#on, 1-‐>2
Backward mo#on: 1<-‐2
Problem Defini#on
t t+1
1) Define regions of interests, or points of interests in the first image at ‘t’ 2) Search for correspondence in the second image at ‘t+1’
Challenges: Image appearance changes, even in the best cases!
Mo#on simplified: G. ScoR
Guy ScoR Ac#ng President of Zambia October 2014 to January 2015
Approaches
Brightness Constancy Based Differen#al Technique, Lucas & Kanade (KLT)
Corner Feature Matching Discrete Matching Technique, ScoR & Longuet-‐Higgins
Takeo Kanade
Differen#al Approach: KLT Tracker
• Detect corners features in first image • Use image patch as feature descrip#on
– Could be extended to color and texture descriptor • Use Lucas-‐Kanade algorithm to compute displacement of the pixels in the patch – Mo#on model could be transla#on (2 dof), affine (6 dof), or more general 3D models
• Subpixel accuracy • Do not need repeated detec#on
Discrete Matching Approach
• Detect corners features in both images • Use image patch as feature descrip#on
– Could be extended to color, texture, SIFT/HOG descriptor
• Find correspondence as Permuta#on (C1, C2) – C1 (in image 1) is the best match to C2 (in image 2) – C2 (in image 2) is the best match to C1 (in image 1)
Discrete Matching Approach
• Detect corners features in both images • Use image patch as feature descrip#on
– Could be extended to color, texture, SIFT/HOG descriptor
• Find correspondence as Permuta#on (C1, C2) – C1 (in image 1) is the best match to C2 (in image 2) – C2 (in image 2) is the best match to C1 (in image 1)
Need to seek Geometrical Valid matching
RANSAC Graph Matching
Object Mo#on Only Compound Mo#on
• Slides from Andrew Cosand
G. L. ScoR, H. C. Longuet-‐Higgins
hRp://www.michaelbach.de/ot/mot_Ternus/index.html
Ternus
Ternus
Ternus
ScoR & Longuet-‐Higgins
Define a distance metric between features Gij=e(-‐rij
2/2σ2)
Create matrix of rela#onships for all possible feature pairs
G11
Gij
Find a Permuta#on P btw points in image 1 to image 2, so that it ‘correlates best’, every point is happy.
t+1
maxP
X
i
X
j
PijGij = trace(PTG)
Singular Value Decomposi#on
SVD factors a matrix into the product of two orthogonal matrices and a diagonal matrix of singular values (eigenvalues).
G = TDU, G is m-‐by-‐n, – T is orthogonal, m-‐by-‐m – D is diag(σ1, σ2, … σp), m-‐by-‐n, p=min{m,n} – U is orthogonal, n-‐by-‐n
ScoR & Longuet-‐Higgins
Use Singular Value Decomposi#on on matrix. G = TDU
ScoR & Longuet-‐Higgins Set diagonal elements of D to 1, ‘recover’ rela#onship matrix. P = TIU = TU
Elimina#ng singular matrix rescales data in feature space, essen#ally sphereing it.
ScoR & Longuet-‐Higgins
Largest feature in row and column indicates mutual best match (correspondence)
Graph matching via SVD
maxP
X
i
X
j
PijGij = trace(PTG)
Goal is to:
The trick is to relax permuta#on P to an orthogonal matrix Q
1) Let F any orthogonal matrix, D a diagonal matrix
2) Transform G to D:
3) Transform solu#on of F by the same transforma#on:
trace(FT ·D) is max. at F = I
D = TT ·G · UT
trace(FTD) = trace(FTTTGUT ) = trace(UTFTTTG) = trace(QTG)
Graph matching via SVD
maxP
X
i
X
j
PijGij = trace(PTG)
Goal is to:
trace(FTD) = trace(FTTTGUT ) = trace(UTFTTTG) = trace(QTG)
Q = T · F · UWhere orthogonal matrix
Since F = I P = T · I · U
ScoR & Longuet-‐Higgins Geometrical Proper#es
• “In several of the examples we have described, and others too numerous to men#on, the circles were derived from the crosses by an affine transforma#on not involving rota#on, and in every case our algorithm succeeds in finding the feature correspondences created by this transforma#on.”
• “Because successive images in a sequence will oven be connected by transforma#ons that are affine or nearly so, this property is one to be welcomed, if not posi#vely required, in a sa#sfactory correspondence algorithm. The following argument is intended to explain why the algorithm performs so well in this respect.”
• “if one set of points in a plane is mapped into another by a transla#on, an expansion or a shear deforma#on, then this 1:1 mapping minimizes the sum of the squares of the distances between corresponding points in the two”
Gij = dist(i, j);
si = Ai · ri + tAffine mo#on:
Discrete Graph World
Con#nuous Geometrical World
mimP
�
i
�
j
PijGij = trace(PT G)
si = ri + t
Assume transla#on mo#on:
Show any other 1:1 mapping results in a greater value for the sum
where i' denotes the new partner of the point i.
�(ri � si�)2
Simple case: transla#on mo#on
The simplest non-‐trivial case (1' = 2,2' = 3,3' =1). The aim is then to show that
si = Ai · ri + t
For general Affine mo#on:
The aim is then to show that:
r · A · r +
A is symmetric and posi#ve definite
Pilu’s Improvement
• Rogue features don’t correspond to anything, complica#ng the process.
• S&LH only deals with proximity and exclusivity.
• Similarity constraint can eliminate rogue features, which shouldn’t be similar to anything.
Pilu’s Improvement
Modify rela#onship metric to include gray-‐level correla#on.
Gij = (e-‐(Cij – 1)2/2γ2) e(-‐rij2/2σ2)
Gij = ((Cij+1) /2) e(-‐rij2/2σ2)
– Adds similarity to feature space (kernel opera#on). – Rogue features can be eliminated because they are not similar to anything.
References – M. Pilu, A direct method for stereo correspondence based on singular value decomposi#on
• variants – G. L. ScoR, H. C. Longuet-‐Higgins, An Algorithm for Associa#ng the Features of Two Images
Forward mo#on, 1-‐>2
Backward mo#on: 1<-‐2
Problem Defini#on
t t+1
1) Define regions of interests, or points of interests in the first image at ‘t’ 2) Search for correspondence in the second image at ‘t+1’
Challenges: Image appearance changes, even in the best cases!
Mo#on simplified: G. ScoR
Approaches
Brightness Constancy Based Differen#al Technique, Lucas & Kanade (KLT)
Corner Feature Matching Discrete Matching Technique, ScoR & Longuet-‐Higgins
Takeo Kanade
Differen#al Approach: KLT Tracker
• Detect corners features in first image • Use image patch as feature descrip#on
– Could be extended to color and texture descriptor • Use Lucas-‐Kanade algorithm to compute displacement of the pixels in the patch – Mo#on model could be transla#on (2 dof), affine (6 dof), or more general 3D models
• Subpixel accuracy • Do not need repeated detec#on
Review:
Op#cal Flow Assump#ons: Brightness Constancy
* Slide from Michael Black, CS143 2003
Op#cal Flow Assump#ons:
* Slide from Michael Black, CS143 2003
Op#cal Flow Assump#ons:
* Slide from Michael Black, CS143 2003
Lucas-‐Kanade tracking Intensity constancy constraint:
Define Sum of Squared Difference, SSD, error as:
Three steps for solving this problem:
Solve for d, warp image, iterate with Newton Raphson.
Step 1
Differen#ate SSD with respect to d,
Differen#ate SSD with respect to d,
Differen#ate SSD with respect to d,
Assume small mo#on, Taylor expansion of J(x+d) is
Step 2
Assume small mo#on, Taylor expansion of J(x+d) is
Combining previous equa#ons…
Combining previous equa#ons…
together…
Two unknown, two linear equa#ons
A: second moment matrix
Error vector b
Error vector b
What if A is not full rank? Recall we compute eigenvalue of A:
[v,d] = eig(A);
diag(d) contains the two eigenvalues, and we want
Edge
– large gradients, all the same – large λ1, small λ2
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
Low texture region
– gradients have small magnitude – small λ1, small λ2
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
High textured region
– gradients are different, large magnitudes – large λ1, large λ2
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
Mo#on simplified: G. ScoR
Iterna#on:
1) Update Ji+1(x) -‐> Ji(x+d) 2) Re-‐compute d, between Ji(x) and I(x)
