Evangelos Loutas and Ioannis Pitas* Dept. of Informatics ...poseidon.csd.auth.gr/LAB_SEMINARS/DigDays/Lectures/... · Aristotle University of Thessaloniki Occlusion Definition Ł

Department of InformaticsDepartment of InformaticsAristotle University of Aristotle University of ThessalonikiThessaloniki

Object TrackingObject Tracking

Evangelos LoutasEvangelos Loutas and and IoannisIoannis PitasPitas**Dept. of InformaticsDept. of Informatics

Aristotle University of Aristotle University of ThessalonikiThessalonikiThessalonikiThessaloniki, GREECE, GREECE

*email: pitas@*email: [email protected]


DefinitionDefinition

Object Tracking: Trace the Object Tracking: Trace the progress of objects or (object progress of objects or (object features) as they move about features) as they move about

in a visual scene. in a visual scene.


Basic concepts neededBasic concepts needed

�� Correlation.Correlation.�� Edge Information.Edge Information.�� Color Information.Color Information.�� SpatioSpatio--temporal information.temporal information.�� Kalman Kalman Prediction.Prediction.


CorrelationCorrelation

�� Given a template T in the form of a small array of image Given a template T in the form of a small array of image intensities find the likely locations of that template in some intensities find the likely locations of that template in some larger test image I.larger test image I.

This means that the mathematical correlation has to be This means that the mathematical correlation has to be maximized.maximized.

dxdyyxTyyxxI ),(),( :Maximize '' ++∫∫


Edge Detection Edge Detection

Edge detection using:Edge detection using:

�� Edge templates.Edge templates.�� LaplacianLaplacian..�� Hough transform.Hough transform.


Extract Color Information Extract Color Information

�� Use the Hue functionUse the Hue function�� Use the Fisher Linear Use the Fisher Linear Discriminant Discriminant function.function.

Fischer(Fischer(II)=)=f.I, If.I, I=(r,g,b)=(r,g,b)


SPATIOSPATIO--TEMPORAL INFORMATIONTEMPORAL INFORMATION

�� Information about objects from single Information about objects from single (spatial) as well as multiple (temporal) (spatial) as well as multiple (temporal) frames/images.frames/images.

�� Previous frames can be used to predict Previous frames can be used to predict object motion. (Motion Prediction) object motion. (Motion Prediction)


Kalman Kalman Filtering(I)Filtering(I)

�� A A Kalman Kalman filter estimates the state of a filter estimates the state of a dynamic system recursively in time, in the dynamic system recursively in time, in the linear minimum mean square error sense, linear minimum mean square error sense, given a time series of vector or scalar given a time series of vector or scalar observations that are linearly related to these observations that are linearly related to these state variables. state variables.


Kalman Kalman Filtering (II)Filtering (II)

�� If the state variables and the noise are If the state variables and the noise are modeled as uncorrelated modeled as uncorrelated Gausian Gausian random random processes then the processes then the Kalman Kalman filter is the filter is the minimum mean square error estimator. minimum mean square error estimator.


Kalman Kalman Filtering Basics(I)Filtering Basics(I)

�� Linear state transition equation :Linear state transition equation :z(k): state vector at time k.z(k): state vector at time k.Φ(Φ(k,kk,k--1): State transition matrix.1): State transition matrix.w(w(k): k): Zero mean white random sequence.Zero mean white random sequence.

Nkkkkkk ,...,1),()1()1,()( =+−−Φ= wzz


Kalman Kalman Filtering Basics(II)Filtering Basics(II)

�� The measurements are related to the state The measurements are related to the state variables as:variables as:

yy(k): Observation vector.(k): Observation vector.HH(k) : Observation matrix.(k) : Observation matrix.vv(k) : Zero mean white observation noise (k) : Zero mean white observation noise

sequencesequenceN1,...,k ),()()()( =+= kkkk vzHy


Algorithm CategorizationAlgorithm Categorization

�� KnowledgeKnowledge--base.base.�� Camera motion.Camera motion.�� Rigid body vs. nonRigid body vs. non--rigid body.rigid body.


Some Major Problem Areas of Object Some Major Problem Areas of Object TrackingTracking

�� Feature SelectionFeature Selection�� OcclusionOcclusion


Feature SelectionFeature Selection--Various Various ApproachesApproaches

�� The need of feature selection.The need of feature selection.�� In general, temporal as well as spatial In general, temporal as well as spatial

variations are used to select features.variations are used to select features.�� The The Kanade Kanade Lucas Lucas Tomasi Tomasi Algorithm Algorithm

approach.approach.�� Active Contour approach.Active Contour approach.


Background SubtractionBackground Subtraction

�� Background subtraction is used for Background subtraction is used for separating moving objects from their separating moving objects from their backgrounds. It is used as a prebackgrounds. It is used as a pre--process in process in advance of feature detection to suppress the advance of feature detection to suppress the background features. The foreground areas background features. The foreground areas are those that satisfy:are those that satisfy:

σ>− ),(),( yxIyxI B


OcclusionOcclusion

DefinitionDefinition�� Occlusion is a set of points that appear in one Occlusion is a set of points that appear in one

image whose corresponding world points are image whose corresponding world points are not visible in another image because an not visible in another image because an opaque object is blocking the view of those opaque object is blocking the view of those points in the other image.points in the other image.


ExamplesExamples


Algorithm ExamplesAlgorithm Examples

�� Active ContoursActive Contours

�� Object tracking using a set of point featuresObject tracking using a set of point features


SnakesSnakes

�� Snake : Deformable curve rSnake : Deformable curve r(s)(s) 00≤≤≤≤≤≤≤≤ss≤≤≤≤≤≤≤≤1.1.�� Maximize F(rMaximize F(r(s)(s)) over ) over 00≤≤≤≤≤≤≤≤ss≤≤≤≤≤≤≤≤1.1.�� The tendency to maximize F is formalized as the The tendency to maximize F is formalized as the

��externalexternal�� potential energy of the dynamical potential energy of the dynamical system.system.

�� The The ��externalexternal�� potential energy is counterbalanced potential energy is counterbalanced by by ��internalinternal�� potential energy.potential energy.


Deformable TemplatesDeformable Templates

�� A parametric shape model rA parametric shape model r(s,(s,XX) ) called called deformable template.deformable template.


Active ContoursActive Contours

�� Newton�s law of motion for a snake with mass Newton�s law of motion for a snake with mass driven by internal and external forces :driven by internal and external forces :

�� ww11, w, w2 2 : Elastic coefficients: Elastic coefficients�� ρ : Mass densityρ : Mass density�� γ : viscous resistance from a medium surrounding the snake.γ : viscous resistance from a medium surrounding the snake.

Fsw

sw

∇+∂

∂+

∂∂

−−= ))()(( 2

22

1 rrrr ttt γρ


Kalman Kalman Filtering Prediction to Active Filtering Prediction to Active ContoursContours

�� The dynamical model is used for prediction.The dynamical model is used for prediction.�� The predicted position is refined using The predicted position is refined using

measured image features.measured image features.�� Form of Dynamical equation :Form of Dynamical equation :

),,(...

wXXX f=


Resistance to occlusionResistance to occlusion

�� The active contours algorithm shows The active contours algorithm shows resistance to partial occlusions. A partial resistance to partial occlusions. A partial occlusion causes loss of measurements . The occlusion causes loss of measurements . The remaining successful observations, together remaining successful observations, together with the dynamical model compensate for with the dynamical model compensate for lost measurements. Observations resume lost measurements. Observations resume after after disocclusiondisocclusion..


Limitations of the traditional modelLimitations of the traditional model

�� Geometrically and topologically simple Geometrically and topologically simple objects can be handled.objects can be handled.

�� The model is inadequate for objects with The model is inadequate for objects with deep cavities or multideep cavities or multi--part objects.part objects.

�� The topology of structure of interest must be The topology of structure of interest must be known in advance in order to define a known in advance in order to define a parametric model.parametric model.


Possible FeaturesPossible Features

�� EdgesEdges�� ValleysValleys�� RidgesRidges


Another ApproachAnother Approach

�� Select the region to be tracked.Select the region to be tracked.�� Define a set of N point features inside the Define a set of N point features inside the

region.region.�� Track the point features using Track the point features using Kanade Kanade --

Lucas Lucas --Tomasi Tomasi algorithm.algorithm.�� Estimate the tracked region. Estimate the tracked region.


Tracking of point featuresTracking of point featuresKanade Kanade -- Lucas Lucas --Tomasi Tomasi approach (I)approach (I)

Find the displacement vector dFind the displacement vector d=[=[ddxx,,ddYY] by ] by minimizing over a window W the dissimilarity minimizing over a window W the dissimilarity between the current and the previous frame:between the current and the previous frame:

xxdxdx dwIJW

)()]2

()2

([ 2−−+= ∫∫ε


Tracking of point featuresTracking of point featuresKanadeKanade -- Lucas Lucas --TomasiTomasi approach (II)approach (II)

�� After setting the derivative equal to zero:After setting the derivative equal to zero:

it is found that in order to perform one it is found that in order to perform one iteration of the minimization the equation :iteration of the minimization the equation :

must be solvedmust be solved

0=∂∂dε

ed =Z


Tracking of point featuresTracking of point featuresKanadeKanade -- Lucas Lucas --TomasiTomasi approach (III)approach (III)

�� Z is a 2x2 matrix depending on the image Z is a 2x2 matrix depending on the image gradient, e is a 2x1 matrix (error vector) gradient, e is a 2x1 matrix (error vector) depending on the frame difference and the depending on the frame difference and the image gradient.image gradient.

�� The final solution is achieved by solving The final solution is achieved by solving repeatedly and shifting I and J repeatedly and shifting I and J image by the computed amount. image by the computed amount.

ed =Z


Feature SelectionFeature Selection

�� The feature selection on The feature selection on Kanade Kanade -- Lucas Lucas --Tomasi Tomasi algorithm is based on the algorithm is based on the requirement that the equation is requirement that the equation is well conditioned.well conditioned.

�� A good feature is defined as one for which A good feature is defined as one for which the matrix Z has two large the matrix Z has two large eigenvalueseigenvalues..

ed =Z


Reliability of Optical FlowReliability of Optical Flow

�� The reliability of optical flow can be defined The reliability of optical flow can be defined as the angle between two lines corresponding as the angle between two lines corresponding to equation to equation .ed =Z


Occlusion Handling in Occlusion Handling in Kanade Kanade --Lucas Lucas -- Tomasi Tomasi AlgorithmAlgorithm

�� Rejection of features whose residue Rejection of features whose residue εε is above is above a certain threshold.a certain threshold.

�� A large residue implies that different motions A large residue implies that different motions exist within the search range of the tracker. exist within the search range of the tracker.


Confronting the Occlusion Problem Confronting the Occlusion Problem (I)(I)

PARTIAL OCCLUSIONPARTIAL OCCLUSIONPredict the motion of occluded features using Predict the motion of occluded features using

motion of the motion of the unoccluded unoccluded features. features.


Confronting the occlusion Confronting the occlusion problem (II)problem (II)

FULL OCCLUSIONFULL OCCLUSION� Find the occluding region.�� Predict the overall occluded region motion usingPredict the overall occluded region motion using

KalmanKalman Filtering.Filtering.�� Determine region Determine region disocclusiondisocclusion..�� Perform a verification procedure.Perform a verification procedure.�� Continue track the Continue track the disoccluded disoccluded region.region.


ResultsResultsResults On Artificial Images

The Region Being Tracked is Found after total occlusion


ResultsResults

“Walker Video Sequence”


ResultsResultsResults On Football Image Sequence (I)

The Region Being Tracked is found after total occlusion and disocclusion.


ResultsResultsResults On Football Image Sequence (II)

The Region Being Tracked is found after total occlusion and disocclusion.


ResultsResultsResults On Football Image Sequence (III)

The Region Being Tracked is found after total occlusion anddisocclusion.


Resistance to partial occlusion(I) Resistance to partial occlusion(I)

The tracker is resistant to partial occlusionThe tracker is resistant to partial occlusion


Resistance to partial occlusion(II) Resistance to partial occlusion(II)


Resistance to partial occlusion(III)Resistance to partial occlusion(III)


BibliographyBibliography

�� Andrew Blake and Michael Andrew Blake and Michael Isard Isard �Active �Active Contours�,Contours�,SpringerSpringer--Verlag Verlag 1998.1998.

�� A. A. Murat Tekalp Murat Tekalp �Digital Video Processing�, �Digital Video Processing�, Prentice Hall 1995.Prentice Hall 1995.

�� C. C. Tomasi Tomasi and T. and T. KanadeKanade, , �Shape and �Shape and Motion from Image Streams: a Factorization Motion from Image Streams: a Factorization Method Method -- Part 3 Detection and Tracking of Part 3 Detection and Tracking of Point Features�, Point Features�,


Bibliography (II)Bibliography (II)

�� Tech. Report CMUTech. Report CMU--CSCS--9191--132, Computer 132, Computer Science Department Carnegie Mellon Science Department Carnegie Mellon University, April 1991.University, April 1991.

�� J. Shi and C. J. Shi and C. TomasiTomasi, , �Good Features to �Good Features to Track�, Track�, IEEE International Conference on IEEE International Conference on Computer Vision and Pattern Recognition Computer Vision and Pattern Recognition (CVPR94), Seattle, June 1994.(CVPR94), Seattle, June 1994.


Bibliography (III)Bibliography (III)

�� Tim Tim McInerney McInerney and and Demetri TerzopoulosDemetri Terzopoulos, , �Topologically Adaptable Snakes�,�Topologically Adaptable Snakes�, Int. Conf. Int. Conf. On Computer Vision (ICCV �95), On Computer Vision (ICCV �95), Cambridge, MA, USA, June 1995.Cambridge, MA, USA, June 1995.

�� RyuzoRyuzo Okada, YoshiakiOkada, Yoshiaki ShiraiShirai and Jun and Jun Miura Miura "Object Tracking based on Optical "Object Tracking based on Optical Flow and Depth",Flow and Depth", Proc. of IEEE/SICE/RSJ Proc. of IEEE/SICE/RSJ Int. Conf. on MFI, pp.565Int. Conf. on MFI, pp.565--571, 1996 571, 1996

Documents

Evangelos Loutas and Ioannis Pitas* Dept. of Informatics ...poseidon.csd.auth.gr/LAB_SEMINARS/DigDays/Lectures/... · Aristotle University of Thessaloniki Occlusion Definition Ł