2011-07-19 Robust Tracking Using Constrains and Context

8/2/2019 2011-07-19 Robust Tracking Using Constrains and Context

1/147

Helmut Grabner

Ftan, 2011/07/19

Robust Tracking using VisualConstrains and Context

ETH-Zurich, Computer Vision Lab 1H. Grabner, Robust Tracking using Visual Constrains and Context


2/147

Put cards on the table: I will NOT talk about



3/147

BUT, I will talk about



4/147

We all know what tracking is, right?

[Grabner et al., VideoProc CVPR06]



5/147

We all know what tracking is, right?

from time t to t+1Time t find again


ac ua o ec pos on


6/147

What to track?



7/147

What to track?

centerpoint



8/147

What to track?

multiplepoints



9/147

Example application: Structure From Motion

[Pollefeyes et al., IJCV 2004]



10/147

What to track?

(body)parts



11/147

Example application: Game interface

[Shotton et al., CVPR 2011]



12/147

What to track?

region



13/147

What to track?

outline



14/147

What to track?

structure



15/147

Example application: Augmented reality

[Wagner et al., ISMAR09]



16/147

In this talk



17/147

OBJECT



18/147

Why is it hard?

initialization



19/147

Why is it hard?

initialization

fast movement



20/147

Why is it hard?

initialization

fast movement

nonlinear dynamics


Wrong prediction

Correctprediction


21/147

Why is it hard?

initialization

fast movement

nonlinear dynamics


mu t p e s m arobjects


22/147

Why is it hard?

initialization

fast movement

nonlinear dynamics


mu t p e s m ar o ects background clutter


23/147

Why is it hard?

initialization

fast movement

nonlinear dynamics



appearance changes

(model drift)


24/147

Why is it hard?

initialization

fast movement

nonlinear dynamics



appearance changes

(model drift) occlusions/ self

occlusions


25/147

In this talk



26/147

OBJECT

CONSTRAINS



27/147

Before we actually start:Two Tracking paradigm



28/147

(i) Traditional tracking looppredict to t+1

time t

measure at t+1

update location

update model



29/147

(ii) Tracking-by-detection

time t time t+1 time t+n

(i) detect object(s)

independently in each frame

(ii) associate detections overtime into tracks



30/147

Using constrains and context

Prediction/Correction(model-free-tracking)

Tracking-by-detection

predict to t+1 measure at t+1

information about the object dynamics

scene context (e.g., geometry)

ETH-Zurich, Computer Vision Lab H. Grabner, Robust Tracking using Visual Constrains andContext

30

time t

update locationupdate model

time t time t+1time t+n

build and adapt the detector

learning (refining) a model during tracking

exploit relations to other objects

object and its surrounding


31/147

Tracking-by-detection

model of the object (object category) is given



32/147

SCENE CONTEXT

[Grabner et al., Photogrammetry & Remote Sensing07]

[Klucker et al., ICCV07 WS on 3dRR]


build and adapt the detector



33/147

How to get the detections?

Nemos Background

Supervised LearningETH-Zurich, Computer Vision Lab 33H. Grabner, Robust Tracking using Visual Constrains and Context


34/147

Fewer Clicks - Less Frustration

Image

Detections

On-linelearning algorithm

updated classifier

Labels



35/147



36/147

Minimize the human labeling effort

Detections

Image

Labels

updated classifier




37/147

Exploiting the 3d structure

Mutliple Images

Depth map



38/147

A 3d teacher

Detections

Image

Labels

updated classifier


verify detections

(no big depth discontinuities)



39/147

Results



40/147

Results



41/147

Results


L 1


42/147

LESSON LEARNED 1

Visual constrains saves

you time!



43/147

SCENE CONTEXT

information about the object dynamics

[Stalder et al., ECCV10]


scene context (e.g., geometry)



44/147



45/147

Detection

Tracking

Input image

Behaviours

Workflow

StatisticsReasoning



46/147

Challenges of an industrial environment



47/147

Challenges of an industrial environment

Occlusions

Self Occlusions

Other movin Ob ects

Similar Objects

Industrial working

conditions

Real-time



48/147

State-of-the-art person detector

i-Lids Scovis-NISSAN

P(p

erson)

ETH-Zurich, Computer Vision Lab 48H. Grabner, Context in Tracking - ETH-Zurich, Computer Vision Lab

H. Grabner, Robust Tracking using Visual Constrains andContext


49/147

reasonablygood detector

Use unlabled datato specialize!



50/147

How to interpret unlabeled data?

Machine Learning?

Transfer learning?

Learning to Learn?



51/147


52/147

Unlabeled data

Thomas Bayes1702-1761


Further Assumptions!

(e.g., manifold assumption,cluster assumption,)


53/147

Additional information/constraints

Geometry and structure Objects are embedded intocontext

Temporal consistency Typical behavior


LESSON LEARNED 2


54/147

LESSON LEARNED 2

Vision is more than pure

machine learning!



55/147

Input to our System

Input image Object detector Confidence map


[Dalal and Triggs, CVPR05]


56/147

Geometric Filter

Confidence map Geometric constrains Confidence map



57/147

Background Filter

Confidence volume Background modeling Confidence volume


[Stauffer and Grimson, CVPR99]


58/147

Trajectory Filter

Confidence volume Vessel-Filtering Confidence volume


[Frangi et al., 98]


59/147

Particle Filter (cont. Trajectories)

Confidence volume Particle filter Confidence volume



60/147

Input

Cascaded ConfidenceFiltering

Geometry Filter

Background Filter

Trajectory Filter



61/147

Results



62/147



63/147


T ki b D t ti


64/147

Input Sequence

Object Detector

Object DetectionsData Assocciation

Tracking-by-Detection

Tracking-by-Detection

[Huang et al., ECCV08]ETH-Zurich, Computer Vision Lab 64H. Grabner, Robust Tracking using Visual Constrains and Context

Object Detections


65/147

Input Sequence

CCF

Object Detections

Data AssocciationTracking-by-Detection

Confidence Map


Improved tracking by detection


66/147

Improved tracking-by-detection



67/147



68/147


69/147


[Leibe et al., PAMI08]


70/147

Model-free-Object Tracking

No object model is given



71/147

LOCAL CONTEXT


time t


object and its surrounding

[Grabner et al., CVPR06, BMVC06]ETH-Zurich, Computer Vision Lab 71H. Grabner, Robust Tracking using Visual Constrains and Context

Objects are not isolated!


72/147

Objects are not isolated!

Learning current object appearance vs. local background.

currentac groun

currentobj. appearance


Tracking as binary classification probelm


73/147


object

background

vs.




74/147


object

background

vs.



75/147

search Region

actual object position

from time t to t+1 evaluate classifier on sub-patches

-

+

- -

-

create confidence mapanalyze map and set new

object positionupdate classifier (tracker)



76/147



77/147

ackin

g

Simple

t



78/147

invis

ibvle

Tr

ackingth


LESSON LEARNED 4


79/147

Foreground/ background

discrimination reducesambiguities.



80/147

Tracking Solved


Does it fail?

If yes, when?

When does it fail


81/147


When does it fail


82/147



83/147

search Region


from time t to t+1 evaluate classifier on sub-patches

-

+

- -

-

create confidence mapanalyze map andset new objectposition

update classifier (tracker)

Self-learning


Drifting


84/147

Tracked Patches Confidence


LESSON LEARNED 5


85/147

Self-learning drifting!



86/147

TRACKING AND DETECTION


time t


learning (refining) a model during tracking

[Grabner et al., ECCV08]ETH-Zurich, Computer Vision Lab 86H. Grabner, Robust Tracking using Visual Constrains and Context

Review: Self-learning


87/147


Tracking is a semi-supervised problem!


88/147

Un-labeled

data


Labeled data

Tracking is a semi-supervised problem!


89/147

Current Model

Fix (initial) ModelPrior


Semi-Supervised Tracking


90/147

Prior


Labeled data

Un-labeleddata

Tracking Loop evaluate classifier on


91/147

Tracking Loop

search Region


evaluate classifier onsub-patches

Prior

create confidence mapupdate classifier (tracker)

??

??


Robust Tacking


92/147


Robust Tracking


93/147


Drifting


94/147

CLICK HERE TO START


LESSON LEARNED 6


95/147

Tracking is a semi-



96/147

Tracking Solved


Does it fail?

If yes, when?


97/147



98/147

Prior is toon ri . irf


to similar objects)


99/147



100/147

Prior is toor ri iv


(e.g., partialocclusions)

LESSON LEARNED 7


101/147

The prior (detector) is-

supervisedmachine learning.



102/147

SCENE CONTEXT


time t


learning (refining) a model during tracking II

[Kalal et al. CVPR10]

[Stalder et al. ICCV09 WS on OLCV]ETH-Zurich, Computer Vision Lab 102H. Grabner, Robust Tracking using Visual Constrains and Context

Review: Self-Learning


103/147

Too adaptive


Review: Semi-Supervised Tracking


104/147

Limit drifting but maybe too

restrictive/general



105/147

Unlabeled data

Thomas Bayes1702-1761


Vision is more than puremachine learning!

LESSON LEARNED 2

Prior Model = Detector


106/147

Non adaptive at all!

Obj. Det.


cf. Tracking-by-DetectionObj. Det.

Adapt the Prior using Visual Constrains!


107/147

new object samples

Obj. Det.


new non-objectsamples

Obj. Det.Background

Object Detectornon adaptive


108/147

Visual ConstrainsBuilding/ Refining amodel usin labeled


-Prior (Detector) with

unlabelled updates

data

Object Trackeradaptive (self-learning)


109/147



110/147



111/147


LESSON LEARNED 8


112/147

Use visual constrains for

detector.


OTHER OBJECTS


113/147

OTHER OBJECTS

redict to t+1 measure at t+1

exploit relations to other objects

[Grabner et al., CVPR11]


time t



Im Carl Track me


114/147


Tracking Carl


115/147


Goal: Estimate the Position of an Object


116/147

Track the Object with occlusions outside the image


Many temporary, but potential verystrong links exists between a tracked


117/147

strong links exists between a trackedobject and parts of the images.

We discover the dynamic elements called SUPPORTERSthat predict the position of the target.


SUPPORTERS

came with different strength.


118/147

change over time.


SUPPORTERS

came with different strength.


119/147

change over time.


SUPPORTERS help Tracking of

bj t hi h h


120/147

objects which changethere appearance very

quickly.

occluded objects or

object outside the image.

small and/or low

textured objects or evenvirtual points.


Local Image Features as Supporters


121/147




122/147




123/147




124/147

Object FeaturesBackgroundFeatures




125/147

Supporters




126/147

Supporters


SUPPORTERS are features which


127/147

contribute to the prediction of the

target position.

They at least temporarily move in away which is statistically related to the

motion of the object.


Discovering the

Supporters


128/147

Predictedpostion

Reliablemeasurments

Model &learning


Reliable Information & Motion Coupling


129/147

x(i)t MotionCoupling

x*

t


Supporter

Strong Supporter

Strong motion coupling

Weak Supporter

Weak motion couplingAlmost Unrelated


130/147

Strong motion coupling

Peaky vote

Weak motion coupling

Blurred voteFeatures


Experimental Results: ETH-CupSequenze


131/147


ETH-Cup: Humans


132/147


ETH-Cup: Of the Web Tracker


133/147


ETH-Cup: Our Result Voting Space


134/147


ETH-Cup: Improving Object Tracking


135/147

10 Humans

( 2 )2 )2 )2 )

Tracker[Stalder et al, OLCV09]

45 % 100 %

+ supporter 89 % 97 %


Please note, a simple interpolation would not work.

ETH-Cup: Supportes

Supporter


136/147

Reliable

Information


Predicted Position(Mode of the Voting Space)

ETH-Cup: Supporters


137/147


Beyond the Image

Voting Space Supporters


138/147


Changing Supporter



139/147


Appearance Change



140/147

Zurich, 2010/06/03 140H. Grabner, Tracking the Invisible - ETH-Zurich, Computer Vision LabETH-Zurich, Computer Vision Lab 140H. Grabner, Robust Tracking using Visual Constrains and Context

Coupled Motion



141/147


Changing Supporters



142/147


Obviously, there are failure cases


. and magician knows that.


143/147


LESSON LEARNED 9


144/147

Look for supporters!

They are common and

take many forms.


OBJECT

Summary & Conclusion


145/147

CONSTRAINS


OBJECT

Summary & Conclusion


146/147

CONSTRAINS


Acknowledgments

Centere of MachineP ti

Computer VisionLaboratory


147/147

Severin StalderLuc Van GoolJiri Matas

PreceptionCzech Technical

University

LaboratoryETH Zurich

Michael Grabner Christian LeistnerHorst Bischof

Institute for Computer Graphics and VisionGraz, University of Technology, Austria


Philippe Cattin

Medical ImageAnalysis Center

University of Basel

Documents

2011-07-19 Robust Tracking Using Constrains and Context