Relations as Context to improve Multi-Target Tracking and Activity Recognition. Cristina Manfredotti 12, Enza Messina 1, David Fleet 2 1 DISCo, University

Relations as Context to improve Multi-Target Tracking and

Activity Recognition.

Cristina Manfredotti12, Enza Messina1, David Fleet2

1DISCo, University of Milano-Bicocca2Computer Science Dept, University of Toronto

C.Manfredotti, E.Messina, D.Fleet: Relations as context. Potsdam, Germany, 14° Sept.09 2

Relations to improve tracking


Complex activity recognition

Y.Ke, R.Sukthankar, M.Hebert; Event Detection in Crowed Videos


Objectives

Goals: 1. To model relations and 2. To maintain beliefs over particular

relations between objects

In order to simultaneously:

• Improve tracking with informed predictions and

• Identify complex activities based on observations and prior knowledge


Relational Domain

Relational Domain: set of objects characterized by attributes1 and with relations1 between them

BoatIdcolorposition(t)velocity(t)direction(t)DecreasingVelocity(t)

A

SameDirection(t)distance(t)Before(t)

Boat BIdcolorposition(t)velocity(t)direction(t)DecreasingVelocity(t)SameDirection(t)distance(t)Before(t)

1Attributes and relations are predicate in FOL.


A Parenthesis:

To model uncertainty in a Relational Domain we will use

Relational (Dynamic) Bayesian Networks


BN: the Alarm example


Thanks to Mark Chavira

A large BN


The Alarm Relational Domain

Relational Domain contains a set of objects with relations between them

Objects

e.g.: Relation

• neighbor• alarm• burglar

• toCall (the howner of the house)• toHear (the alarm)

neighbor’s attributes: capacity of hearing, attention, ...

alarm’s attributes: its volume, its sensibility, ...


Alarm RBN:

Alarm.Volume

NeighborCalls

Earthquacke

Neigh.DegOfDef

Neigh.NoiseAround


Closing the parenthesis ...

• Syntax RBN:– a set of nodes, one for each variable

– a directed, acyclic graph – a conditional distribution for each node

given its parents

This distribution must take into account the actual “complexity” of the nodes!

• Syntax RBN:– a set of nodes, one for each predicate

– a directed, graph– a conditional distribution for each node

given its parents,


Relational State

The State of a Relational Domain is the set of the predicates that are true in the Domain.

r

a

s

ss

Relational state

State of attributes

State of relations


Dynamics

The State of a Relational Domain is the set of the predicates that are true in the Domain.

State evolves with time

We extend a RBN to a RDBN as we are used to extend a BN to a DBN.


Inference

Markov assumption andConditional independence of data on state.

bel(st) = ® p(zt|st)s p(st|st-1)bel(st-1)dst-1

Bayesian Filter

The problem of computing:

bel(st) = p(st|z1:t)


p(xt-1|z1:t-1) p(xt|z1:t-1) p(xt|z1:t)

Bel(xt-1) Bel(xt) Bel(xt)

Transition model

Sensor model

t = t+1

~

Transition model

Sen

sor

mo

del

Inference

Relations in the State result in correlating the State of different objects between them

p(xt-1|z1:t-1) p(xt|z1:t-1) p(xt|z1:t)

Bel(xt-1) Bel(xt) Bel(xt)

Transition model

Sensor model

t = t+1


Sensor model (1st assumption)

part of the state relative to relations, sr, not directly observable

p(zt|st) = p(zt|sa

t)

observation zt independent by the relations between objects.

Intuitively:

Travelling Together vs Being Close


Transition model: a trick

p(st|st-1) = p(sat,sr

t|sat-1, sr

t-1)

Sat-1

Srt-1

Sat

Srt

Intuitive


p(sat,sr

t|sat-1,sr

t-1)=

But srt independent by sa

t-1 given srt-1 and sa

t

p(sat,sr

t|sat-1,sr

t-1) = p(sat|sa

t-1,srt-1) p(sr

t|srt-1, sa

t)

bel(st) = p(st|z1:t) = p(sat,sr

t|z1:t)

bel(st)=αp(zt|sat,sr

t)s p(sat,sr

t|sat-1,sr

t-1)bel(st-1)dst-1

p(zt|sat,sr

t) = p(zt|sat)

Relational Inference

p(sat|sa

t-1,srt-1) p(sr

t|sat-1,sr

t-1, sat)

Transition model (2nd assumption)


Conditional Probability Distribution

FOPT: a Probabilistic Tree whose nodes are FOL formulas

CPD relationt(x,y):

relationt-1(x,y)

p(relationt(x,y))

xt, yt

CPD yt:

x, relationt-1(x,y)

p(yt|yt-1)

TF

p(yt|yt-1) p(xt|xt-1,yt-1,rt-1)


* It is a technique that implements a recursive Bayesian Filter through a Monte Carlo simulation. The key idea is to represent the posterior pdf as a set of samples (particles) paired with weights and to filter the mesurament based on these weights..

Particle Filtering* (general case)


Relational Particle Filter


RPF: extraction

Xat,(m)

Xrt,(m)

Xat,(m)

~ p(xat,(m)|sa

t-1,srt-1)

Xat,(m)

~ p(xrt,(m)|sa

t = xat,(m),sr

t-1)

Xrt,(m)


RPF: weighting

The consistency of the probability function ensures the convergence of the algorithm.

Xat,(m)

Xrt,(m)

Weight ( ) ~p(zt|xat)

The weighting step is done according to the attributes part of each particle only, the relational part follows.


Tracking AND activity Recognition

Xat,(m)

Xrt,(m)

Xat,(m)

Xrt,(m)

Xat,(m)

Xa{t,(m)}Xo{t,(m)}

Xrt,(m)

Xat+1,(m)

1° step of sampling: prediction of the state of attributes

Xat,(m)

Xa{t,(m)}Xo{t,(m)}

Xrt,(m)

Xat+1,(m)

Xa{t,(m)}Xo{t,(m)}

Xrt+1,(m)

2° step of sampling: prediction of the state of relationsOr activity prediction


Canadian Harbor: rendezvous

Same speed


Canadian Harbor: Avoidance

Constant speed


Exp: attributes’transition


Exp: relations’transition


Exp: Results

method TP ratio TN ratio Mean Tracking Error (km)

RPF 0.4545 0.7235 1.8379

PF 3.3906

random 0.4444 0.4841


To conclude ...

• Modeling Relations “dynamically”:– To improve multi target tracking– To recognize complex activities

• Inference in Dynamic Relational Domain– In theory complex BUT

– Simplified by

• “smart decomposition” of the transition model

• “non-relational” sensor model

• Results are promising

Documents

Relations as Context to improve Multi-Target Tracking and Activity Recognition. Cristina Manfredotti 12, Enza Messina 1, David Fleet 2 1 DISCo, University