Robust Multi-Pedestrian Tracking in Thermal-Visible

Surveillance Videos

Alex Leykin, Yang Ran, and Riad Hammoud

GoalCreate a pedestrian tracker that operates in:

1. Varying illumination conditions2. Crowded environment

To achieve it we create a fusion pedestrian tracker that uses input from:

3. IR camera4. RGB camera

Our approach consists of three stages:

BG Subtraction Bayesian tracker Pedestrian Classifier

Background Model

Two stacks of codeword values (codebooks)

Color• μRGB

• Ilow • Ihi

Thermal• thigh

• tlow

codeword

codebook

Adaptive Background Update

If there is no match create new codeword

Else update the codeword with new pixel information

If >1 matches then merge matching codewords

I(p) > Ilow

I(p) < Ihigh

(RGB(p)∙ μRGB) < TRGB

t(p)/thigh > Tt1

t(p)/tlow > Tt2

Match pixel p to the codebook b

Subtraction ResultsColor model only

Combined color and thermal model

Tracking

Location of each pedestrian is estimated probabilistically based on:

Current image Model of pedestrians Model of obstacles

The goal of our tracking system is to find the candidate state x` (a set of bodies along with their parameters) which, given the last known state x, will best fit the current observation z

P(x’| z, x) = P(z|x’) · P(x’|x)

observation likelihood state prior probability

Tracking – Accepting the State

))|'(

)'|(

)(

)(,1min(),(

xxtm

xxtm

xP

xPxx

x’ and x candidate and current states

P(x) stationary distribution of Markov chain

mt proposal distribution

Candidate proposal state x’ is drawn with probability mt(x’|x) and then accept it with the probability α(x, x’)

Tracking: Priors

N(hμ, hσ2) and N(wμ,wσ

2) body width and height

U(x)R and U(y)R body coordinates are weighted uniformly within the rectangular region R of the floor map.

d(wt, wt−1) and d(ht, ht−1) variation from the previous size

d(xt, x’t−1) and d(y, y’t−1) variation from Kalman predicted position

N(μdoor, σdoor) distance to the closest door (for new bodies)

Constraints on the body parameters:

Temporal continuity:

Tracking Likelihoods: Distance weight plane

2hPz

Problem: blob trackers ignore blob position in 3D (see Zhao and Nevatia CVPR 2004)

Solution: employ “distance weight plane” Dxy = |Pxyz, Cxyz| where P and C are world coordinates of the camera and reference point correspondingly and

Tracking Likelihoods: Z-buffer

0 = background, 1=furthermost body, 2 = next closest body, etc

Tracking: Likelihoods

),(11 1 ttcolorcolor ccBwP

I

DZOIP xyZ

)( )0(

O

DIZOP xyZ

)( )0(

Implementation of z-buffer (Z) and distance weight plane (D) allows to compute multiple-body configuration with one computationally efficient step.

Let I - set of all blob pixels O - set of body pixels

Then

Color observation likelihood is based on the Bhattacharya distance between candidate and observed color histograms

Tracking: Jump-Diffuse Transitions

Add a new body Delete a body Recover a recently deleted body Change body dimensions

Change body position (optimize with mean shift)

Tracking: Anisotropic Weighted Mean Shift

Classic Mean-Shift Our Mean-Shift

t-1

t

H

t

Tracking Results

Sequence

number

Frames

People

People

missed

False hits

Identity switches

1 1054

15 3 1 3

2 0601

8 0 0 0

3 1700

16 5 1 2

4 1506

3 0 0 0

5 2031

2 0 0 0

6 1652

4 0 0 0

%% 8544

48 12.5

4.1 10.4

Finding Gait in Spatio-temporal Space

Periodic Pattern Grouping Theory: A two-dimensional pattern that

repeats along one dimension is called a frieze pattern in the mathematics and geometry literature

Group theory provides a powerful tool for analyzing such patterns

Mapping gait into repetitive texture Translational symmetry: Class P4 Detection: verifying spatio-temporal

texture Localization: extract orientation

(trajectory), frequency (period), representative motif (signature)

Symmetries of the gait patterns

Classifying PedestriansX-t Image Extract Lattice Signature Results

Finding Gait in Spatio-temporal Space

Details in Y. Ran, I. Weiss, Q. Zheng, and L. S. Davis. Pedestrian detectionvia periodic motion analysis. IJCV 2007

Classification Results

Tracking results

Pedestrian Detection

Contributions

Robust to illumination changes

Resolving track initialization ambiguity with MCMC

Non-unique body-blob correspondence

Gait detector runs in real time

Future Work Extend binary background mask with

foreground probability values

Incorporate these probabilities into appearance-based fitness equation for particle filter-based tracker

Utilize tracklet stitching (via particle tracker) to decrease the number of broken paths

Aknowledgements

Organizers of OTCBVS Benchmark Dataset Collection

http://www.cse.ohio-state.edu/otcbvs-bench

Thank you!

alexleykin.zapto.org