Motion Segmentation By Hadas Shahar (and John Y.A.Wang, and Edward H. Adelson, and Wikipedia and...

Motion Segmentation

By Hadas Shahar

(and John Y.A.Wang, and Edward H. Adelson, and Wikipedia and YouTube)

Introduction

•When given a video input, we would like to divide it into segments according to the different movement types.

•This is useful for object tracking and video analysis

Session Map

•Building Blocks:▫Layered Image representation▫Optic Flow Estimation▫Affine Motion Estimation

•Algorithm Walkthrough•Examples

Session Map

Layered Image Representation

•Given a simple movement, what would be the best way to represent it?

•Which parameters would you select to represent for the following scene?

Layered Image RepresentationFor any movement we would like to have 3 maps:

•The Intensity Map

•The Alpha Channel (or- opacity)

•The Warp Map (or- optic flow)

For example, if we take a scene of a hand moving over a background, we would like to get:

•Given these maps, it’s possible to easily reproduce the occurring movement.

•But how can we produce these maps?

Session Map

Optic Flow Estimation (this is the heavy part)

•The Optical Flow is a field of vectors describing the movement in the image

•For example:

•Note! Optical Flow doesn’t describe the occurring movement, but the movement we perceive.

Look at the Barber’s pole for example

• The actual motion is RIGHT

• But the perceived motion (or- Optical Flow) is UP

Optic Flow Estimation- the Lucas-Kanade method

In order to identify movements correctly, we have to work with several assumptions:

•Brightness Consistency- the movement won’t affect the brightness of the object

•Constant Motion in a neighborhood- neighboring pixels will move together

Definitions:

X(t) is the point X at time t (X=x,y)

I(X(t)) is the brightness of point X at time t.

is the gradient.

Brightness Consistency Assumption:

I(x(t),t)= const for any t

Meaning- the brightness of point x(t) is constant.Therefore the time derivative must be 0:

We would like to focus on this part, the velocity

( ( ), ) 0Td dx II x t t I

dt dt t

But why is the Intensity assumption not enough?

Let’s look at the following example and try to determine the optical flow:

It looks like the grid is moving down and to the right

But it can actually be one of the following:

Since our window of observation is too small, we can’t infer the actual motion taking place.

This is called the Aperture Problem

And this is why we need the 2nd constraint

Constant Motion In a Neighborhood:

We assume the velocity is the same in our entire window of observation-

( ', ) ( ', ) 0 ' ( )T II x t V x t x W x

W(x) is the window or- environment, of x

There is a trade off here-

The larger the window the less accurately it represents the velocity (since we assume the velocity is constant there)

And in the other direction- the smaller the window the more likely we are to have the aperture problem

Sadly, since there are some changes in intensity (due to environment changes or even sensor noise) –the derivative will never actually be 0.

So, we take the least square error:2

( ) | ( ', ) ( ', ) |Tt

E V I x t V I x t

The minimal value will occur when the 2nd derivative is 0:

So V is:

2 2 0dE

MV qdV

M I I ( )

1V M q

A few notes regarding M:M is a 2x2 matrix -made up of the gradient times its transpose:

We can divide M into 3 cases (this is going to be very similar to the Harris corner detection)

x x yM

Case1: If the gradient is 0, M=0,there are no eigenvalues and V can have any value.

This occurs when our window is at a clear region:

Case2: If the gradient is constant, M is not 0 but we’ll receive only 1 eigenvalue.

This occurs when our window is at an edge:

Case1: If M is invertible (det=0), we can find V easily

This occurs when our window is at a corner:

After we find V for every window, we get Velocity vector map, or- the Optical Flow.

Session Map

Affine Estimation

•In Affine Estimation, we assume our motions can be described by affine transformations.

•This includes:▫Translations▫Rotations▫Zoom▫Shear

And this does cover a lot of the motions we encounter in the real world

Affine Estimation

The idea behind Affine Estimation is quite simple-

Find the affine transformation between 2 images, that will have the minimal difference.

Affine Estimation

Quick reminder:

Affine Estimation

There are several ways to do this, most commonly by matching feature-points between the 2 images and calculating the Affine transformation matrix(remember?)

What we’ll use won’t be based on feature points, but on the Velocity vector calculated from the Optical Flow.

We’ll get to that later though, so for now- no formulas!

Session Map

Part 2- The Algorithm Walkthrough•So how can we combine all the

information we gathered so far into creating our 3 maps for every frame?

The Algorithm Walkthrough

•Here’s the basic idea:

First, we calculate the Optical Flow- this gives us the Warp map.

But since it will only look for 1 overall motion, it may disregard object boundaries and we’ll get several different objects in our motion.

Optical Flow Estimator

Then, we divide the image(s) into arbitrary sub-regions, and use Affine Estimation, which helps us find the local motions within every sub-region

Affine Regression and Clustering

Then we check the difference between our initial guess and the movement observed.

And reassign the sub-regions to minimize the error

Our estimation using an affine transformation

Actual change

Hypothesis Testing

We repeat the cycle iteratively, constantly refining the motion estimation.

Convergence is achieved when either:1. Only a few points

are reassigned in each iteration

2. Max number of iterations is reached

Region reassignments- in each iteration we refine our estimation results

This segmentation is what provides us with the Opacity Map

Reminder- This is an Affine Transformation matrix:

Made up of 6 variables, to cover the rotation, translation, zoom and shear operations

The Algorithm Walkthrough- definitions

Let V be our Velocity (obtained by the Optical Flow estimation)

We would like to use the velocity to represent the Affine Transformation:

But how can we work with V in such a way?We break V into Vx and Vy, 2 vectors representing the velocity in the X and Y direction respectively

0 0 1 1

a b c x

V d e f y

Vx(x,y)= [aX, bY, c]Vy(x,y)= [dX, eY, f]

Where a,b,c,d,e,f are the variables of the affine transformation

•Let be the ith hypothesis vector.

Meaning- is the affine transformation matrix we believe would best represent the ith region’s movement.

•We would like to break H into its x and y parts as well:

[ , ]i yi xiH H H

[ , , ]

[ , , ]xi xi xi xi

yi yi yi yi

H a b c

H d e f

And last but not least-

We define

[ , ,1]T x y

*That’s our original coordinates vector

So basically so far we got the following parameterization:

V Hi Φ

Then we can define our affine equations like this:

V x y H

And we can calculate Hi from V using the following formula:

[ , ] ([ ( , ), ( , )] ) Tyi xi y x

H H V x y V x y

This is the Velocity x and y parameters

This is the pseudo inverse matrix

Summed over all the pixels in the region

We know we can divide our region segmentations into 2 cases:

•A sub-region contains several object boundaries

(ie- the region contains several small movements)

•An object is covered by several sub-regions

(ie- we need to merge regions in order to view the full movement)

Case 1- sub-region contains several object boundaries

In this case, since we divide our image into fairly small regions, we would like to ignore these sections.

These regions will have a large residual error, so we can identify them and remove them from our calculations

Case 2- An object is covered by several sub-regions

In this case, we would like to merge the 2 (or more) sub-regions, so they would cover our entire object.

Since the sub-regions contains the same moving object, its movement parameters will be very similar.

So how do we do it?

We move our Hypotheses into affine motion space-Parameterizing them using the velocity rather than the spatial values.

Then we group them usingK-Means Clustering.(we already know how to do that!)

This merges similar Hypotheses and provides us with a single representative for each motion.

Now that we calculated the affine transformation for each region, we would like check how we did compared to the actual movement.For this we use a mean square cost function:

Where:i(x,y) is the sub-region assigned to that x,y coordinateV(x,y) is the estimated motion fieldVHi(x,y) is the affine motion field of the ith hypothesis

( ( , )) ( ( , ) ( , ))Hix y

C i x y V x y V x y

We wish to minimize the difference between our hypothesis and the actual motion, so for each sub-region we’ll take the minimum value:

( ( , )) ( ( , ) ( , ))Hix y

C i x y V x y V x y

20 ( , ) arg min[ ( , ) ( , )]Hii x y V x y V x y

i0 is the minimum cost assignment- the minimal value for the ith region

Now we divide the image(s) into motion regions by taking a threshold on the i0 values as:

01 ( , )( , )

if i x y HP x y

otherwise

This gives us the opacity map!

And then we just iterate until the regions stop changing, or until the max number of iterations is reached.

And we’re done!

Examples!• https://www.youtube.com/watch?v=7BtlB8rEqrY

• https://www.youtube.com/watch?v=nnp9qc8O8eE

• https://www.youtube.com/watch?v=4ny8rR1hesU

Summary

We saw how to calculate the Optical Flow in a given video, and how to use the Optical Flow in combination with the Affine estimation model iteratively to get better approximation of the motion.

Conclusions

Motion segmentation is an important part of any motion related algorithm, and is a useful and powerful tool for in computer vision.

Credits

•John Y.A. Wang & Edward H. Adelson- Layered Representation for Motion Analysis (1993)

•Edward H. Adelson- Layered Representation for Image Coding (1991)

•Lucas B. & Kanade T.- Optical flow algorithm

Motion Segmentation By Hadas Shahar (and John Y.A.Wang, and Edward H. Adelson, and Wikipedia and...

Documents

Happiness Venezuela - Tal Ben Shahar

MBLT15: Shahar Waiser, Gett

Shahar Slides on Machine learning

Bellini - Adelson e Salvinib

Catalogo Toke de Hadas

Imagenes De Hadas

Yuval Shahar, M.D., Ph.D

Bassem Al-Tamimi v Adelson Complaint

Summit 2010 Infra Services Shahar

CATALOGO HADAS

Adelson v. DiPaola, 1st Cir. (1997)

Shahar palace

Manual Reiki Hadas

Tal Ben-Shahar - EVE

David Hadas AbstractedNetworks[1]

Hotel shahar palace

TAL BEN-SHAHAR, Ph.D

SHAHAR Project History - UCL - London's Global … SHAHAR COMPONENT SHAHAR Project History: Learning from experience in urban programming CARE - Bangladesh in collaboration International

Eyal Shahar, MD, MPH Professor (credit to Doron Shahar, BS) December 5, 2013

ANGELES HADAS Y DUENDES.pdf