An Object Tacking Paradigm with Active Appearance Models for Augmented Reality

Presented by Pat Chan Pik Wah

28/04/2005

Qualifying Examination

Outline

Research Objective Introduction

Augmented Reality Object Tracking Active Appearance Models (AAMs)

Proposed Object Tracking Paradigm Paradigm Architecture Experiments

Research Issues Conclusion

Research Objective

Object tracking is an essential component for Augmented Reality.

There is a lack of good object tracking paradigm. Active Appearance Models is promising. Propose a new object tracking paradigm with AAMs

in order to provide a real-time and accurate registration for Augmented Reality.

Nature of the paradigm: Effective Accurate Robust

Augmented Reality

An Augmented Reality system supplements the real world with virtual objects that appear to coexist in the same space as the real world

Properties : Combine real and virtual objects in a real environment Runs interactively, and in real time Registers (aligns) real and virtual objects with each other

Augmented Reality

Projects related to AR

Augmented Reality

Display Presenting virtual objects on real environment

Tracking Following user’s and virtual object’s movements

by means of a special device or techniques 3D Modeling

Forming virtual object Registration

Blending real and virtual objects

Object Tracking

Visual content can be modeled as a hierarchy of abstractions.

At the first level are the raw pixels with color or brightness information.

Further processing yields features such as edges, corners, lines, curves, and color regions.

A higher abstraction layer may combine and interpret these features as objects and their attributes.

Pixels

edges, corners, lines, curves, and color regions

Object

Object Tracking

Accurately tracking the user’s position is crucial for AR registration

The objective is to obtain an accurate estimate of the position (x,y) of the object tracked

Tracking = correspondence + constraints + estimation Based on reference image of the object, or properties

of the objects. Two main stages for tracking object in video:

Isolation of objects from background in each frames Association of objects in successive frames in order to trace

them

Object Tracking

Object Tracking can be briefly divides into following stages: Input (object and camera) Detecting the Objects Motion Estimation Corrective Feedback Occlusion Detection

Object Tracking

Expectation Maximization Find the local maximum likelihood solution Some variables are hidden or incomplete

Kalman Filter Optimal linear predict the state of a model

Condensation Combines factored sampling with learned dynam

ical models propagate an entire probability of object position

and shape

Object Tracking

Pervious Work : Marker-based Tracking Feature-based Tracking Template-based object tracking Correlation-based tracking Change-based tracking 2D layer tracking tracking of articulated objects

Pervious Work

Marker-based Tracking Marker-less based Tracking Feature-based Tracking

Shape-based approaches Color-based approaches

Pervious Work

Template-based object tracking Fixed template matching

Image subtraction Correlation

Deformable template matching

Pervious Work

Object tracking using motion information Motion-based approaches Model-based approaches Boundary-based approaches

Snakes Geodesic active contour models

Region-based approaches

Active Appearance Models

The Active Appearance Model (AAM) algorithm is a powerful tool for modeling images of deformable objects.

AAM combines a subspace-based deformable model of an object’s appearance

Fit the model to a previously unseen image.

Timeline for development of AAMs and ASMs

Active Appearance Models (AAMs)

2D linear shape is defined by 2D triangulated mesh and in particular the vertex locations of the mesh.

Shape s can be expressed as a base shape s0.

pi are the shape parameter. s0 is the mean shape and the matrices si are the eigenvectors corresp

onding to the m largest eigenvalues

Active Appearance Models (AAMs)

The appearance of an independent AAM is defined within the base mesh s0. A(u) defined over the pixels u ∈ s0

A(u) can be expressed as a base appearance A0(u) plus a linear combination of l appearance

Coefficients λi are the appearance parameters.

A0(u) A1(u) A2(u) A3(u)

Active Appearance Models (AAMs)

The AAM model instance with shape parameters p and appearance parameters λ is then created by warping the appearance A from the base mesh s0 to the model shape s.

Piecewise affine warp W(u; p):(1) for any pixel u in s0 find out which triangle it lies in,(2) warp u with the affine warp for that triangle.

M(W(u;p))

Fitting AAMs

Minimize the error between I (u) and M(W(u; p)) = A(u).

If u is a pixel in s0, then the corresponding pixel in the input image I is W(u; p).

At pixel u the AAM has the appearance

At pixel W(u; p), the input image has the intensity I (W(u; p)).

Minimize the sum of squares of the difference between these two quantities:

uu u u

DEMO Video – 2D AAMs

DEMO Video – 2D AAMs

Recent Work for Improving AAMs

Combine 2D+3D AAMs

Combined 2D + 3D AAMs

At time t, we have 2D AAM shape vector in all N images into a matrix:

Represent as a 3D linear shape modes W = MB =

Compute the 3D Model

AAM shapes

AAM appearance

First three 3D shapes modes

Constraining an AAM with 3D Shape

Constraints on the 2D AAM shape parameters p = (p1, … , pm) that force the AAM to only move in a way that is consistent with the 3D shape modes:

and the 2D shape variation of the 3D shape modes over all imaging condition is:

Legitimate values of P and p such that the 2D projected 3D shape equals the 2D shape of AAM. The constraint is written as:

An Object Tacking Paradigm with Active Appearance Models

Proposed Object Tracking Paradigm

Training Active Appearance Model

Training Images

1. Shape Model2. Appearance Model Initialization Motion Modeling Kalman Filter

Occlusion Detection

VideoVideo

Paradigm Architecture

Steps in Object Tracking Paradigm

Preporcessing Training the Active Appearance Model. Get the shape model and the appearance model for the object to

be tracked. Initialization

Locating the object position in the video. In our scheme, we make use of AAMs.

Motion Modeling Estimate the motion of the object Modeling the AAMs as a problem in the Kalman filter to perform t

he prediction. Occlusion Detection

Preventing the lost of position of the object by occluding of other objects.

Enhancing Active Appearance Models

Shape

Appearance

Combine the shape and the appearance parameters for optimization

In video, shape and appearance may not enough, there are many characteristics and features, such as lightering, brightness, etc…

L=[L1, L2, ……, Lm]T

Iterative Search for Fitting Active Appearance Model

Iterative Search for Fitting Active Appearance Model

Can be improved by:1. Prediction matrix2. Searching space

Initialization for AAMs

Motion Modeling

Initial estimate in a frame should be better predicted than just the adaptation from the previous frame.

Can be achieved by motion estimation AAMs can do the modeling part Kalman filter can do the prediction part

Kalman Filter

Adaptive filter Model the state of a discrete dynamic

system. Originally developed in 1960 Filter out noise in electronic signals.

Kalman Filter

Formally, we have the model

For our tracking system,

1

1

1 1

1

( )

( )

( )

k

k

n

n n

n

k

k k k

k

k

k k

k

S

x A u

c

S

x B A u w

c

S

z H A u v

c

1

1

1

1

1

'

'

1 0 0

0 1 0

0 0 1 0 '

0 0 0 1 '

1 0 0 0

0 1 0 0

'0 0 1 0

'0 0 0 1

k

k

k

k

n

nn

n

n

k k

k

kk k

k

k

x

yx

x

y

xtyt

x wx

y

x

yz v

x

y

11

1

1 0

0 1

1 0

0 1

nn

n

kk k

k

kk k

k

xx

y

xx w

y

xz v

y

Kalman Filter

Occlusion Detection WHY?WHY?

Positioning of objects To perform cropping When a real object overlays a virtual one, the

virtual object should be cropped before the overlay HOW?HOW?

High resolution and sharp object boundaries Right occluding boundaries of objects Camera matrix for video capturing

Proposed Object Tracking Paradigm

Training Active Appearance Model

Training Images

1. Shape Model2. Appearance Model Initialization

Active Appearance Model Fitting

Kalman Filter

Occlusion Detection

VideoVideo

Paradigm Architecture

Experimental Setup

AAM-api from DTU OpenCV Pentium 4 CPU 2.00GHz and 512MB

RAM

Experiment on AAMs (1)

Training Image

Experiment on AAMs (1)

Shape Texture

Experiment on AAMs (1)

Initialization After optimized

Demo Video

Demo Video

Demo Video

Demo Video

Experiment on AAMs (2)

Training Images

Experiment on AAMs

Shape Texture

Experiment on AAMs

Trapped in local minimum

Initialization After optimized

Experiment on AAMs

Experiment on AAMs

Fit to the face

Initialization After optimized

Experiment on AAMs

Object Tracking with AAMs

Experiment on Kalman Filter

Demo Video

Experiment on Kalman Filter

Demo Video

Research Issues

AAMs tracking is accurate Very slow Cannot perform real-time tracking

Kalman filter help is to increase the speed in prediction Modeling the problem from AAMs to Kalman Filter

Improving the fitting algorithm in the AAMs Occlusion detection

Important to object tracking Preventing the lost of the position

Conclusion

We have done a survey on object tracking and active appearance model is done

We proposed a paradigm on video object tracking with active appearance models

Goal: Robust Real-time Good performance

We have done some initial experiments: Experiments on AAMs Experiments on Kalman filter for object tracking

Q & A