Periodic Motion Detection via Approximate Sequence Alignment

Ivan Laptev*, Serge Belongie**, Patrick Perez*

*IRISA/INRIA, Rennes, France

**Univ. of California, San Diego, USA

May 9-10, 2005

Motivation

Dominant motion estimation Works very well for a panning camera and static backgrounds Problems for scenes with motion parallax and multiple motions

Motivation

Dominant motion estimation Works very well for a panning camera and static backgrounds Problems for scenes with motion parallax and multiple motions

Motivation

Assumptions about background motion

e.g. Homography:

Target == Outlier

Assumptions about target motionDetect motion of specific type (Target == Inlier)

- more difficult than for dominant motion

- may have advantages in complex scenes

Here: the type of target motion --- periodic

Periodic motion Periodic views can be approximately treated as

stereopairs

Periodic motion Periodic views can be approximately treated as

stereopairs

Fundamental matrix is generally time-dependent

Periodic motion estimation ~ sequence alignment

Sequence alignment

Generally hard problem Unknown positions and motions of cameras Unknown temporal offset Possible time warping

Prior work treats special cases Caspi and Irani “Spatio-temporal alignment of sequences”,

PAMI 2002 Rao et.al. “View-invariant alignment and matching of video

sequences”, ICCV 2003 Tuytelaars and Van Gool “Synchronizing video sequences”,

CVPR 2004

Useful in Reconstruction of dynamic scenes Recognition of dynamic scenes

Assumptions

Constant translation Assume the camera is translating with velocity relatively to

the object

For sequences

corresponding points are related by

All corresponding periodic points are on the same epipolar line

Points with non-constant motion

Rejects trivial case of pure translation Can be detected by maximizing local variation of space-time

gradients (Laptev and Lindeberg, IJCV 2005)

Space-time interest points

c1

c2

c3

c4

Second-momentmatrix

Local maxima of H over (x,y,t)

Detected points

Points with similar

neighborhoods

Periodic motion detection

1. Corresponding points have similar descriptors

2. Same period for all features

3. Spatial arrangement of features across periods satisfy epipolar constraint:

Use RANSAC to estimate F and p

Periodic motion detection

Original space-time features RANSAC estimation of F,p

Periodic motion detection

Original space-time features RANSAC estimation of F,p

Periodic motion detection

Original space-time features RANSAC estimation of F,p

Periodic motion detection

Original space-time features RANSAC estimation of F,p

Periodic motion segmentation Assume periodic objects are planar

Periodic points can be related by a dynamic homography:

linear in time

Periodic motion segmentation Assume periodic objects are planar

Periodic points can be related by a dynamic homography:

RANSAC estimation of H and p

linear in time

Object-centered stabilization

Segmentation

Disparity estimation

Graph-cut segmentation

Segmentation

Segmentation

Conclusion

Direct method for periodic motion detection and segmentation

Constant translation assumption might be relaxed by tracking the modeling H and F as non-linear matrix functions

Extension to non-periodic motion recognition via sequence alignment using corresponding space-time points

Segmentation