Embed Size (px)
Citation preview
Differential Structure of Movies
• How can we describe the local structure of an image sequence?
• We will assume that a movie is a smooth function of 2+1 dimensional space-time
• Looking for generic properties
Approaches for Motion Analysis
• Optical flow• Spatio-temporal texture• Spatio-temporal differential invariants
Optical Flow
• Geometry of the projected motion of particles in the observers field of view
• Binding hypothesis needed to use it on image sequences
• “Top down”• Local formulation, but non-local, due to binding
hypothesis• Undefined when particles appears and disappears,
e.g. motion boundaries• Does not use image structure
Spatio-Temporal Differential Invariants
• Local geometry of spatio-temporal images• ”Bottom Up”, low level• No binding hypotheses, connection to the
environment considered as a higher level problem
• Well defined everywhere• Does use image structure, extension of low
level vision for still images
Overview
• Galilean geometry• Moving frames• Image geometry• 1+1 dimensional Galilean differential invariants• 2+1 dimensional Galilean differential invariants• What is required for a more realistic movie model
Galilean Geometry
• Spatial and temporal translation a, spatial rotation R and spatio-temporal shear v
nSORvaatxvR
tx n
,,,10
x
t
x
t
Shear
Galilean Geometry
• Insensitive to constant relative motion for parallel projection, approximately otherwise
• Simplest meaning full model• Assumed implicitly when one talk about optical
flow invariants: div, rot, dev, i.e. first order flow • Shape properties from the environment can be
derived from relative motion• (Newton physics describe Galilean invariants)
Galilean Invariants
• Planes of simultaneity (constant t) are invariant and has Euclidean geometry: distances and angles are invariants– i.e. an image sequence
• The temporal distance between planes of simultaneity is an invariant
0 if,,
,,,
tyxp
tyxptp
S
T
Galilean ON-System
• An n+1 dimensional Galilean ON-system (e1,e2,,e0) is s.t. (e1,e2,,en) is an Euclidean ON-system and ||e0||T=1
Moving Frames
• Galilean geometry has no metric• We will use Cartan's method of moving
frames, that does not require a metric• Moving frame: e:M! G½ GL(n)• Attach a frame that is adapted to the local
structure in each point• Differential geometry: the local change of
the frame: de
Moving Frames
1
1
ABACACABC
eACedAAdAide
Aie
C(A) contains the differential geometric invariants expressed in the global frame i
Image Geometry
• Image space: E2I - trivial fiber bundle with Euclidian base space and log intensity as fiber (Koenderink 02) z=f(x,y)
• f is smooth• Image geometry
– Global gray level transformations– Lightness gradients
Gradient Gauge
• For points where rf0 we can choose an adapted ON-frame {u,v} s.t. fu=0
• All functions over iuj
v f, i+j¸ 1 becomes invariants w.r.t. rotation in space and translation in intensity
Aiffff
f y
x
yx
xy
v
u
1
Gray Level Invariants
tionstransformaintensity monotonicand spacein rotation w.r.t.invariant
curvaturegradient -curvature isophote -
,
,0
0
12
12
12
2212
12
12
dvdudvffdu
ffc
cc
ffdyffffdxffff
c
cc
AC
v
uv
v
uu
v
u
yx
xyyyyxxxyxyx
0,0
0,0
Hessian Gauge
• For points whererf0 we can choose an ON-frame {p,q} s.t. fpq=0 and |fpp|>|fqq|
• All functions over ipj
q f, i+j¸ 2 becomes invariants w.r.t. rotation in space, translation in intensity and addition of a linear light gradient (Koenderink 02)
xxyy
xy
y
x
q
p
fff
Ai
2tan,cossinsincos
Galilean 1+1 D
• Two cases:– Isophotes cut the spatial line, motion according
to the constant brightness assumption– Isophotes are tangent to the spatial line (along
curves), creation, annihilation
Tangent Gauge
• Let {t,x} be a global Galilean ON-frame, for points where fx0 we can define an adapted Galilean ON-frame {s,x} s.t. fs=0.
.10
1
,0,
Aiff
fff
x
txt
x
s
xts
xts
Isophote Invariants
divergence- onaccelerati-
,00
0
x01
s01
01
01
cca
dxdsa
dxffds
ffc
cAC
x
sx
x
ss
0,0
a
0,0
a
Hessian Gauge
• Let {t,x} be a global Galilean ON-frame, for points where fxx0 we can define an adapted Galilean ON-frame {r,x} s.t. frx=0.
.10
1
,0,
Aiff
ffff
x
txxtx
x
r
xxtxxrrx
xtr
Hessian Invariants
divergence- onaccelerati-
,00
0
x01
r01
01
01
cca
dxdra
dxffds
ffc
cAC
xx
rxx
xx
rrx
0,0
a
0,0
a
Galilean 2+1 D
• General case• Also here are two different main cases
– Isophote surfaces transversal to the spatial plane. Motion of isophote curves in the image
– Isophote surfaces tangent to the plane. Creation, annihilation and saddle points
Invariants in the General Case
au, av - accelerationu, v - divergenceu, v - skew of the ”flow field” - rotation of the plane in the temporal directionu, v - flow line curvature in the plane
∂s
∂u
∂v=1 v x v y
0 cos −sin 0 sin cos ∂t
∂x
∂ y=Ai ,
C A=0 au dsu du u dv av ds v duv dv0 0 ds u duv dv0 −ds u du v dv 0
More Descriptive Invariants
• D - rate of strain tensor for the spatio-temporal part of the frame field
• a, curl D, div D, def D - are flow field invariants• a, , , u, v - are not flow field invariants
a=au2av2 , a=arctan av /au ,
D=u u
v v =u−v
2 0 1−1 0uv
2 1 00 11
2 u−v uv
u v v−u = curlD
2 0 1−1 0divD
2 1 00 1defD
2Q −11 0
0 −1Q
Tangent Gauge
• Let {t, x, y} be a global Galilean ON-frame, for points where ||{fx,fy}||0 we can define an adapted Galilean ON-frame {s,u,v} s.t. fs=fu=fsu=0
• Principal acceleration extrema
• Direction of u constant along s – used in Guichard (98)
Tangent Gauge
1
22
100010
1
00
00
0011
BABCBCBAC
BAiff
ff
ffff
fffffff
Aiffff
ff
v
u
tv
t
uu
tu
uuv
uvt
v
u
s
uvuutusu
vts
vuts
y
x
t
yx
xy
yxv
u
t
uu
suv
uuv
uvsv
vsvv
uusuuu
vssv
uu
ssu
uuv
uvssu
vv
uu
ff
ffff
ffff
ffaff
ffffa
dvducdvdsac
dvdudsac
cccc
BAC
12
02
01
12
12
0201
0000
0
Hessian Gauge
• Let {t, x, y} be a global Galilean ON-frame, we define an adapted Galilean ON-frame {r, p, q} s.t. fpq= frp= frq=0.
• Also defined when the spatial tangent disappears, e.g. for creation and disappearance of structure
• r is the same vector field as when the optical flow constraint equation is solved for the spatial image gradient
Hessian Gauge
1
22
2
2tan
cossin0sincos0001
1
100010
11
0
0
BABCBCBAC
fff
BAiff
Aiffffffff
fff
ffff
ffff
xxyy
xy
y
x
r
yxq
p
r
y
x
txxtyxytxyytxxyty
xyyyxxy
x
r
yyxytyry
xyxxtxrx
yxtr
)22/(
0000
0
12
02
01
12
12
0201
qqpprpq
qqrpqq
pprpqp
qqrqqq
pprppp
qqrrqq
pprrpp
qp
qqq
ppp
fff
ff
ff
ff
ff
ffa
ffa
dqdpc
dqdpdrac
dqdpdrac
cccc
BAC
Real Image Sequences
• Localized filters are not invariant w.r.t. Galilean shear, velocity adapted (s – directed) filters are needed
• More generic singular cases for imprecise measurements
Conclusion
• Theory about differential invariants for smooth Galilean spatio-temporal image sequences
• Local operators• “Bottom up”• Contains more information about the image
sequence than optical flow• Extension of methods for still images
