Relational Dynamic Bayesian Networks to improve Multi-Target Tracking. Cristina Manfredotti and Enza...

Relational Dynamic Bayesian Networks to improve

Multi-Target Tracking.

Cristina Manfredotti and Enza Messina

DISCo, University of Milano-Bicocca

2C.Manfredotti, E.Messina: RDBNs to improve MTT. Mercure Chateau Chartrons, Bordeaux, France, Sept. 28 - Oct 2, 2009

Relations to improve tracking

3C.Manfredotti, E.Messina: RDBNs to improve MTT. Mercure Chateau Chartrons, Bordeaux, France, Sept. 28 - Oct 2, 2009

Complex activity recognition

Y.Ke, R.Sukthankar, M.Hebert; Event Detection in Crowed Videos

4C.Manfredotti, E.Messina: RDBNs to improve MTT. Mercure Chateau Chartrons, Bordeaux, France, Sept. 28 - Oct 2, 2009

Objectives

Goals: 1. To model relations and 2. To maintain beliefs over particular

relations between objects

In order to simultaneously:

• Improve tracking with informed predictions and

• Identify complex activities based on observations and prior knowledge

5C.Manfredotti, E.Messina: RDBNs to improve MTT. Mercure Chateau Chartrons, Bordeaux, France, Sept. 28 - Oct 2, 2009

Relational Domain

Relational Domain: set of objects characterized by attributes1 and with relations1 between them

CarIdcolorposition(t)velocity(t)direction(t)DecreasingVelocity(t)

A

SameDirection(t)distance(t)Before(t)

Car BIdcolorposition(t)velocity(t)direction(t)DecreasingVelocity(t)SameDirection(t)distance(t)Before(t)

1Attributes and relations are predicate in FOL.

6C.Manfredotti, E.Messina: RDBNs to improve MTT. Mercure Chateau Chartrons, Bordeaux, France, Sept. 28 - Oct 2, 2009

Relational State

The State of a Relational Domain is the set of the predicates that are true in the Domain.

r

a

s

ss

Relational state

State of attributes

State of relations

7C.Manfredotti, E.Messina: RDBNs to improve MTT. Mercure Chateau Chartrons, Bordeaux, France, Sept. 28 - Oct 2, 2009

Relational Bayesian Networks:

Uncertainty in a Relational Domain Relational (Dynamic) Bayesian Networks

• Syntax RBN:– a set of nodes, one for each variable

– a directed, acyclic graph – a conditional distribution for each node

given its parents

This distribution must take into account the actual “complexity” of the nodes!

• Syntax RBN:– a set of nodes, one for each predicate

– a directed, graph– a conditional distribution for each node

given its parents

8C.Manfredotti, E.Messina: RDBNs to improve MTT. Mercure Chateau Chartrons, Bordeaux, France, Sept. 28 - Oct 2, 2009

Dynamics

The State of a Relational Domain is the set of the predicates that are true in the Domain.

State evolves with time

We extend a RBN to a RDBN as we are used to extend a BN to a DBN.

9C.Manfredotti, E.Messina: RDBNs to improve MTT. Mercure Chateau Chartrons, Bordeaux, France, Sept. 28 - Oct 2, 2009

Inference

Markov assumption andConditional independence of data on state.

bel(st) = ® p(zt|st)s p(st|st-1)bel(st-1)dst-1

Bayesian Filter

The problem of computing:

bel(st) = p(st|z1:t)

10C.Manfredotti, E.Messina: RDBNs to improve MTT. Mercure Chateau Chartrons, Bordeaux, France, Sept. 28 - Oct 2, 2009

Inference

Relations in the State result in correlating the State of different objects between them

p(xt-1|z1:t-1) p(xt|z1:t-1) p(xt|z1:t)

Bel(xt-1) Bel(xt) Bel(xt)

Transition model

Sensor model

t = t+1

11C.Manfredotti, E.Messina: RDBNs to improve MTT. Mercure Chateau Chartrons, Bordeaux, France, Sept. 28 - Oct 2, 2009

Sensor model (1st assumption)

part of the state relative to relations, sr, not directly observable

p(zt|st) = p(zt|sa

t)

observation zt independent by the relations between objects.

Intuitively:

Travelling Together vs Being Close

12C.Manfredotti, E.Messina: RDBNs to improve MTT. Mercure Chateau Chartrons, Bordeaux, France, Sept. 28 - Oct 2, 2009

Transition model: a trick

p(st|st-1) = p(sat,sr

t|sat-1, sr

t-1)

Sat-1

Srt-1

Sat

Srt

Intuitive

13C.Manfredotti, E.Messina: RDBNs to improve MTT. Mercure Chateau Chartrons, Bordeaux, France, Sept. 28 - Oct 2, 2009

p(sat,sr

t|sat-1,sr

t-1)=

But srt independent by sa

t-1 given srt-1 and sa

t

p(sat,sr

t|sat-1,sr

t-1) = p(sat|sa

t-1,srt-1) p(sr

t|srt-1, sa

t)

bel(st) = p(st|z1:t) = p(sat,sr

t|z1:t)

bel(st)=αp(zt|sat,sr

t)s p(sat,sr

t|sat-1,sr

t-1)bel(st-1)dst-1

p(zt|sat,sr

t) = p(zt|sat)

Relational Inference

p(sat|sa

t-1,srt-1) p(sr

t|sat-1,sr

t-1, sat)

Transition model (2nd assumption)

14C.Manfredotti, E.Messina: RDBNs to improve MTT. Mercure Chateau Chartrons, Bordeaux, France, Sept. 28 - Oct 2, 2009

* It is a technique that implements a recursive Bayesian Filter through a Monte Carlo simulation. The key idea is to represent the posterior pdf as a set of samples (particles) paired with weights and to filter the mesurament based on these weights..

Particle Filtering* (general case)

15C.Manfredotti, E.Messina: RDBNs to improve MTT. Mercure Chateau Chartrons, Bordeaux, France, Sept. 28 - Oct 2, 2009

Relational Particle Filter

16C.Manfredotti, E.Messina: RDBNs to improve MTT. Mercure Chateau Chartrons, Bordeaux, France, Sept. 28 - Oct 2, 2009

RPF: extraction

Xat,(m)

Xrt,(m)

Xat,(m)

~ p(xat,(m)|sa

t-1,srt-1)

Xat,(m)

~ p(xrt,(m)|sa

t = xat,(m),sr

t-1)

Xrt,(m)

17C.Manfredotti, E.Messina: RDBNs to improve MTT. Mercure Chateau Chartrons, Bordeaux, France, Sept. 28 - Oct 2, 2009

RPF: weighting

The consistency of the probability function ensures the convergence of the algorithm.

Xat,(m)

Xrt,(m)

Weight ( ) ~p(zt|xat)

The weighting step is done according to the attributes part of each particle only, the relational part follows.

18C.Manfredotti, E.Messina: RDBNs to improve MTT. Mercure Chateau Chartrons, Bordeaux, France, Sept. 28 - Oct 2, 2009

Experiments: FOPT

19C.Manfredotti, E.Messina: RDBNs to improve MTT. Mercure Chateau Chartrons, Bordeaux, France, Sept. 28 - Oct 2, 2009

Experiments: Transition Model

• If relation true

• If relation false

20C.Manfredotti, E.Messina: RDBNs to improve MTT. Mercure Chateau Chartrons, Bordeaux, France, Sept. 28 - Oct 2, 2009

Experiments: Results

21C.Manfredotti, E.Messina: RDBNs to improve MTT. Mercure Chateau Chartrons, Bordeaux, France, Sept. 28 - Oct 2, 2009

Further experiments

Data: 15 simulated objects.From each cell, an object can jump to one of the n next cells where n depends by the cell.

Objects can move together. If traveling together,

two (or more) objects will always be in cells from which it is possible for one to reach the

other or vice-versa.

If traveling together, two objects will behave similarly (i.e. if one

turns left, the other will follow).

22C.Manfredotti, E.Messina: RDBNs to improve MTT. Mercure Chateau Chartrons, Bordeaux, France, Sept. 28 - Oct 2, 2009

Tracking AND activity Recognition

Xat,(m)

Xrt,(m)

Xat,(m)

Xrt,(m)

Xat,(m)

Xa{t,(m)}Xo{t,(m)}

Xrt,(m)

Xat+1,(m)

1° step of sampling: prediction of the state of attributes

Xat,(m)

Xa{t,(m)}Xo{t,(m)}

Xrt,(m)

Xat+1,(m)

Xa{t,(m)}Xo{t,(m)}

Xrt+1,(m)

2° step of sampling: prediction of the state of relationsOr activity prediction

23C.Manfredotti, E.Messina: RDBNs to improve MTT. Mercure Chateau Chartrons, Bordeaux, France, Sept. 28 - Oct 2, 2009

step 12step 24

True Positive Rate

False P

ositive Rate

The worst (time step 24) and the best (time step 12) ROC curve for the relation recognition task.

Further Results

0 1

1

24C.Manfredotti, E.Messina: RDBNs to improve MTT. Mercure Chateau Chartrons, Bordeaux, France, Sept. 28 - Oct 2, 2009

PF: 4.6500 2.2333 3.7333 2.7667 …RPF: 4.6000 2.4667 1.3333 2.1000 …

PF : 4.7000 3.6667 5.6667 2.6000 … RPF: 4.6000 3.5333 5.2667 2.5333 …

Further Results (cont.)

Tracking error (distance) for each of the 15 objects.

Comparable behaviour of the errors BUT

for related objects RPF trackes always better than PF.

PF : 4.6667 4.6667 3.8333 1.9333 … RPF: 4.7667 2.7667 3.5333 1.5333 …

PF: 2.0667 5.9000 1.6000RPF: 2.0333 5.8333 2.2333

25C.Manfredotti, E.Messina: RDBNs to improve MTT. Mercure Chateau Chartrons, Bordeaux, France, Sept. 28 - Oct 2, 2009

To conclude ...

• Modeling Relations “dynamically”:– To improve multi target tracking– To recognize complex activities

• Inference in Dynamic Relational Domain– In theory complex BUT

– Simplified by

• “smart decomposition” of the transition model

• “non-relational” sensor model

• Showed promising results