Readings
• Szeliski,R.Ch.7• Bergenetal.ECCV92,pp.237-252.• Shi,J.andTomasi,C.CVPR94,pp.593-600.• Baker,S.andMaPhews,I.IJCV2004,pp.221-255.
• SlideCredits:Szeliski,ShahandB.Freeman
2
Recap:Es8ma8ngOp8calFlow
3
• Assumetheimageintensityisconstant
( )tyxI ,, ( )dttdyydxxI +++ ,,=
I
Time=t Time=t+dt
FirstAssump8on:BrightnessConstraint
4
( )tyxI ,, ( )dttdyydxxI +++ ,, !I(x(t) + u.�t, y(t) + v.�t)� I(x(t), y(t), t) ⇡ 0
AssumingIisdifferen8ablefunc8on,andexpandthefirsttermusingTaylor’sseries:
@I
@x
dx
dt
+@I
@y
dy
dt
+@I
@t
= 0
Ix
u+ Iy
v + It
= 0Compactrepresenta8on
Brightnessconstancyconstraint
SecondAssump8on:GradientConstraint
5
Lecture10:Mo8onModels,FeatureTracking,andAlignment
Velocityvectorisconstantwithinasmallneighborhood(LUCASANDKANADE)
E(u, v) =
Z
x,y
(Ix
u+ I
y
v + I
t
)2dxdy
@E(u, v)
@u=
@E(u, v)
@v= 0
2(Ix
u+ Iy
v + It
)Ix
= 0
2(Ix
u+ Iy
v + It
)Iy
= 0
Recap:Lucas-Kanade
6
Lecture10:Mo8onModels,FeatureTracking,andAlignment
PI2x
PIx
IyP
Ix
Iy
PI2y
� uv
�= �
PIx
ItP
Iy
It
�
Txx
Txy
Txy
Tyy
� uv
�= �
Txt
Tyt
�StructuralTensorrepresenta8on
u =Tyt
Txy
� Txt
Tyy
Txx
Tyy
� T 2xy
and v =Txt
Txy
� Tyt
Txx
Txx
Tyy
� T 2xy
Piaalls&Alterna8ves
• Brightnessconstancyisnotsa8sfied– Correla8onbasedmethodcouldbeused
• Apointmaynotmovelikeitsneighbors– Regulariza8onbasedmethods
• Themo8onmaynotbesmall(Taylordoesnothold!)– Mul8-scalees8ma8oncouldbeused
7
Lecture10:Mo8onModels,FeatureTracking,andAlignment
Mul8-ScaleFlowEs8ma8on
8
Lecture10:Mo8onModels,FeatureTracking,andAlignment
imageIt-1 imageI
GaussianpyramidofimageIt GaussianpyramidofimageIt+1
imageIt+1imageItu=10pixels
u=5pixels
u=2.5pixels
u=1.25pixels
Recap:Horn&Schunck
• Globalmethodwithsmoothnessconstrainttosolveapertureproblem
• Minimizeaglobalenergyfunc8on
• Takepar8alderiva8vesw.r.t.uandv:
9
Lecture10:Mo8onModels,FeatureTracking,andAlignment
E(u, v) =
Z
x,y
[(Ix
u+ I
y
v + I
t
)2 + ↵
2(|ru|2 + |rv|2)]dxdy
(Ix
u+ Iy
v + It
)Ix
� ↵2ru = 0
(Ix
u+ Iy
v + It
)Iy
� ↵2rv = 0
GlobalMo8onModels(Parametric)Allpixelsareconsideredtosummarizeglobalmo8on!• 2DModels
– Affine– Quadra8c– Planarprojec8ve(homography)
• 3DModels– Inst.Cameramo8onmodels– Homography+epipole– Plane+parallax
10
Mo8onModels
11
Lecture10:Mo8onModels,FeatureTracking,andAlignment
Translation
2 unknowns
Affine
6 unknowns
Perspective
8 unknowns
3D rotation
3 unknowns
GlobalMo8on
12
EsOmatemoOonusingallpixelsintheimage
GlobalMo8on
13
EsOmatemoOonusingallpixelsintheimage
GlobalMoOoncanbeusedto• Removecamera
mo8on• Object-based
segmenta8on• generatemosaics
GlobalMo8on
14
EsOmatemoOonusingallpixelsintheimage
GlobalMoOoncanbeusedto• Removecamera
mo8on• Object-based
segmenta8on• generatemosaics
Recap:ObjectTracking
• Trackanobjectoverasequenceofimages
15
ChallengesinObjectTracking
16
ChallengesinObjectTracking
• Whichfeaturestotrack?
17
ChallengesinObjectTracking
• Whichfeaturestotrack?• Efficienttracking
18
ChallengesinObjectTracking
• Whichfeaturestotrack?• Efficienttracking• Appearanceconstraintviola8on• …
19
ChallengesinObjectTracking
• Whichfeaturestotrack?• Efficienttracking• Appearanceconstraintviola8on• …
20
Shi-TomasiFeatureTracker
• GoodFeaturestoTrack
21
Shi-TomasiFeatureTracker
• GoodFeaturestoTrack– FindgoodfeaturesusingeigenvaluesofHessianmatrix(thresholdonthesmallesteigenvaluewhencompu8ngHarriscornerdetec8on)
22
Shi-TomasiFeatureTracker
• GoodFeaturestoTrack– FindgoodfeaturesusingeigenvaluesofHessianmatrix(thresholdonthesmallesteigenvaluewhencompu8ngHarriscornerdetec8on)
– TrackfromframetoframewithLK
23
Shi-TomasiFeatureTracker
• GoodFeaturestoTrack– FindgoodfeaturesusingeigenvaluesofHessianmatrix(thresholdonthesmallesteigenvaluewhencompu8ngHarriscornerdetec8on)
– TrackfromframetoframewithLK– Checkconsistencyoftracksby“affineregistra8on”tothefirstobservedinstanceofthefeature
24
Shi-TomasiFeatureTracker
25
ShiandTomasiCVPR1994GoodFeaturesToTrack.
KLTTracking
• KLT:Kanade-Lucas-Tomasi
26
KLTTracking
• KLT:Kanade-Lucas-Tomasi• Trackingdealswithes8ma8ngthetrajectoryofanobjectintheimageplaneasitmovesaroundascene
27
KLTTracking
• KLT:Kanade-Lucas-Tomasi• Trackingdealswithes8ma8ngthetrajectoryofanobjectintheimageplaneasitmovesaroundascene
• Objecttracking(car,airplane,person)
28
KLTTracking
• KLT:Kanade-Lucas-Tomasi• Trackingdealswithes8ma8ngthetrajectoryofanobjectintheimageplaneasitmovesaroundascene
• Objecttracking(car,airplane,person)• Featuretracking(Harriscorners)
29
KLTTracking
• KLT:Kanade-Lucas-Tomasi• Trackingdealswithes8ma8ngthetrajectoryofanobjectintheimageplaneasitmovesaroundascene
• Objecttracking(car,airplane,person)• Featuretracking(Harriscorners)• Mul8pleobjecttracking
30
KLTTracking
• KLT:Kanade-Lucas-Tomasi• Trackingdealswithes8ma8ngthetrajectoryofanobjectintheimageplaneasitmovesaroundascene
• Objecttracking(car,airplane,person)• Featuretracking(Harriscorners)• Mul8pleobjecttracking• Trackinginsingle/mul8plecamera(s)
31
KLTTracking
• KLT:Kanade-Lucas-Tomasi• Trackingdealswithes8ma8ngthetrajectoryofanobjectintheimageplaneasitmovesaroundascene
• Objecttracking(car,airplane,person)• Featuretracking(Harriscorners)• Mul8pleobjecttracking• Trackinginsingle/mul8plecamera(s)• Trackinginfixed/movingcamera 32
KLTTrackingAlgorithm• FindGoodFeaturesToTrack
– HarrisCorners(thresholdedonsmallesteigenvalues)
• UseLKalgorithmtofindop8calflows• UseCoarse-to-Finestrategytodealwithlargemovements
• Whencrea8nglongtracks,checkappearanceofregisteredpatchagainstappearanceofini8alpatchtofindpointsthathavedrited
33
RecentDevelopmentsatOp8calFlow
• StartwithLKorsimilarmethods+ Gradientconsistency+ Energyminimiza8onwithsmoothingterm+ Regionmatching+ KeyPointmatching
34Large displacement optical flow, Brox et al., CVPR 2009
Region-based +Pixel-based +Keypoint-based
RecentDevelopmentsatOp8calFlow• UseofMachineLearning
– DeepLearning(ICCV2015,Fischeretal.,FlowNet)
35
DeepFlow(LargeDisplacementOp8calFlow)• Basicallyitisamatchingalgorithmwithvaria8onalapproach
[Weinzaepfeletal.,ICCV2013].
• Densecorrespondence(matching)• Self-smoothmatching• Largedisplacementop8calflow
– hPps://www.youtube.com/watch?v=k_wkDLJ8lJE
9/22/16
36
• CanweuseSIFTfeaturesfortracking?
9/22/16
37
Lecture8:Mo8onModels,FeatureTracking,andAlignment
Ex:SIFTTracking
38
à Frame 0 Frame 100
Howtoevaluatecorrectnessofop8calflows?
9/22/16
39
Lecture8:Mo8onModels,FeatureTracking,andAlignment
Op8calFlow-Quan8ta8veEvalua8on
• Whereu=(u,v)iscomputed,u=(u*,v*)groundtruthvelocityvectors.
40
Eep2 =p
(u� u⇤)2 + (v � v⇤)2
Eep1 = |u� u⇤|+ |v � v⇤|
Eang = arccos (
uTu⇤
|u||u⇤| )
Interpreta8onofOp8calFlowFields
41
Interpreta8onofOp8calFlowFields
42
ObjectfeaturesallhaveZerovelocity.
Interpreta8onofOp8calFlowFields
43
Interpreta8onofOp8calFlowFields
44
ObjectismovingtotheRight.
Interpreta8onofOp8calFlowFields
45
Interpreta8onofOp8calFlowFields
46
ObjectismovingDirectlytowardthecamerathatissta8onary
Interpreta8onofOp8calFlowFields
47
Interpreta8onofOp8calFlowFields
48
Cameraismovingintothescene,andanobjectmovingpassedthecamera
Interpreta8onofOp8calFlowFields
49
Interpreta8onofOp8calFlowFields
50
Objectisrota8ngaboutthelineofsighttothecamera
Interpreta8onofOp8calFlowFields
51
Interpreta8onofOp8calFlowFields
52
Objectisrota8ngaboutanaxisperpendiculartothelineofsight.
Applica8oninImageAlignment
• Mo8oncanbeusedforimagealignment
53⎥⎦
⎤⎢⎣
⎡⎥⎦
⎤⎢⎣
⎡=⎥
⎦
⎤⎢⎣
⎡
yx
dcba
yx''
Pixelloca8onsat8metPixelloca8onsat8met+1
Prac8ce:HomogenousCoordinates
54
y
x
tyytxx
+=
+=
''
Q:HowcanwerepresenttranslaOonasa3x3matrix?
Prac8ce:HomogenousCoordinates
55
y
x
tyytxx
+=
+=
''
Q:HowcanwerepresenttranslaOonasa3x3matrix?
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
=
1001001
y
x
tt
ranslationT
Prac8ce:HomogenousCoordinates
56
y
x
tyytxx
+=
+=
''
Q:HowcanwerepresenttranslaOonasa3x3matrix?
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
=
1001001
y
x
tt
ranslationT⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
+
+
=
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
=
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
111001001
1''
y
x
y
x
tytx
yx
tt
yx
tx=2ty=1
Prac8ce:Basic2DTransforma8ons
57
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
ΘΘ
Θ−Θ
=
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
11000cossin0sincos
1''
yx
yx
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
=
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
11001001
1''
yx
tt
yx
y
x
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
=
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
11000101
1''
yx
shsh
yx
y
x
Translate
Rotate Shear
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
=
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
11000000
1''
yx
ss
yx
y
x
Scale
AffineTransforma8on
58
• Affinetransforma8onsarecombina8onsof…– Lineartransforma8ons,and– Transla8ons
• Proper8esofaffinetransforma8ons:– Origindoesnotnecessarilymaptoorigin– Linesmaptolines– Parallellinesremainparallel– Ra8osarepreserved– Closedundercomposi8on– Modelschangeofbasis
⎥⎥
⎦
⎤
⎢⎢
⎣
⎡
⎥⎥
⎦
⎤
⎢⎢
⎣
⎡=
⎥⎥
⎦
⎤
⎢⎢
⎣
⎡
wyx
fedcba
wyx
100''
⎥⎥
⎦
⎤
⎢⎢
⎣
⎡
⎥⎥
⎦
⎤
⎢⎢
⎣
⎡=
⎥⎥
⎦
⎤
⎢⎢
⎣
⎡
wyx
ihgfedcba
wyx
'''
projec8ve
AffineTransforma8on
59
⎥⎥
⎦
⎤
⎢⎢
⎣
⎡
⎥⎥
⎦
⎤
⎢⎢
⎣
⎡=
⎥⎥
⎦
⎤
⎢⎢
⎣
⎡
wyx
fedcba
wyx
100''
⎥⎥
⎦
⎤
⎢⎢
⎣
⎡
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
⎥⎥
⎦
⎤
⎢⎢
⎣
⎡
⎥⎥⎦
⎤
⎢⎢⎣
⎡ΘΘΘ−Θ
⎥⎥
⎦
⎤
⎢⎢
⎣
⎡=
⎥⎥
⎦
⎤
⎢⎢
⎣
⎡
wyx
sysx
tytx
wyx
1000000
1000cossin0sincos
1001001
'''
p’ = T(tx,ty) R(Θ) S(sx,sy) p
Affinematrixdecomposi8onTranslaOon+rotaOon+scaling
Ques8ons?
• PA2willbeincludingop8calflowes8ma8ons.
9/22/16
60
Lecture8:Mo8onModels,FeatureTracking,andAlignment