Motion from image and inertial measurements

(additional slides)

Dennis Strelow

Carnegie Mellon University

Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 2

Outline

Robust image feature tracking (in detail)

Lucas-Kanade and real sequences

The “smalls” tracker

Motion from omnidirectional images

Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 3

Robust image feature tracking: Lucas-Kanade and real sequences (1)

Combining image and inertial measurements improves our situation, but…

we still need accurate feature tracking tracking

some sequences do not come with inertial measurements

Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 4

Robust image feature tracking: Lucas-Kanade and real sequences (2)

better feature tracking for improved 6 DOF motion estimation

remaining results will be image-only

Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 5

Robust image feature tracking: Lucas-Kanade and real sequences (3)

Lucas-Kanade has been the go-to feature tracker for shape-from-motion

minimizes a correlation-like matching error

using general minimization

evaluates the matching error at only a few locations

subpixel resolution

Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 6

Robust image feature tracking: Lucas-Kanade and real sequences (4)

Additional heuristics used to apply Lucas-Kanade to shape-from-motion:

task: heuristic:

choose features to track high image texture

identify mistracked, occluded, no-longer-visible

convergence, matching error

handle large motions image pyramid

Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 7

Robust image feature tracking: Lucas-Kanade and real sequences (5)

But Lucas-Kanade performs poorly on many real sequences…

Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 8

Robust image feature tracking: the “smalls” tracker (1)

smalls is a new feature tracker targeted at 6 DOF motion estimation

exploits the rigid scene assumption

eliminates the heuristics normally used with Lucas-Kanade

SIFT is an enabling technology here

Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 9

Robust image feature tracking: the “smalls” tracker (2)

First step: epipolar geometry estimation

use SIFT to establish matches between the two images

get the 6 DOF camera motion between the two images

get the epipolar geometry relating the two images

Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 10

Robust image feature tracking: the “smalls” tracker (3)

Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 11

Robust image feature tracking: the “smalls” tracker (4)

Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 12

Robust image feature tracking: the “smalls” tracker (5)

Second step: track along epipolar lines

use nearby SIFT matches to get initial position on epipolar line

exploits the rigid scene assumption

eliminates heuristic: pyramid

Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 13

Robust image feature tracking: the “smalls” tracker (6)

Third step: prune features

geometrically inconsistent features are marked as mistracked and removed

clumped features are pruned

eliminates heuristic: detecting mistracked features based on convergence, error

Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 14

Robust image feature tracking: the “smalls” tracker (7)

Fourth step: extract new features

spatial image coverage is the main criterion

required texture is minimal when tracking is restricted to the epipolar lines

eliminates heuristic: extracting only textured features

Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 15

Robust image feature tracking: the “smalls” tracker (8)

Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 16

Robust image feature tracking: the “smalls” tracker (9)

left: odometry only right: images only

average error: 1.74 m

maximum error: 5.14 m

total distance: 230 m

Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 17

Robust image feature tracking: the “smalls” tracker (10)

Recap:

exploits the rigid scene and eliminates heuristics

allows hands-free tracking for real sequences

can still be defeated by textureless areas or repetitive texture

Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 18

Outline

Robust image feature tracking (in detail)

Motion from omnidirectional images

Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 19

Motion from omnidirectional images (1)

Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 20

Motion from omnidirectional images (2)

Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 21

Motion from omnidirectional images (3)

Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 22

Motion from omnidirectional images (4)

Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 23

Motion from omnidirectional images (5)

left: non-rigid camera right: rigid camera

squares: ground truth points solid: image-only estimates

dash-dotted: image-and-inertial estimates

Dennis Strelow -- Motion estimation from image and inertial measurements – January 6, 2005 24

Motion from omnidirectional images (6)

In this experiment:

omni images

conventional images + inertial

have roughly the same advantages

But in general:

inertial has some advantages that omni images alone can’t produce

omni images can be harder to use