Rita Cucchiara, Simone Calderara, Francesco Solera

Imagelab

DIPARTIMENTO DI INGEGNERIA «Enzo Ferrari»

Università di Modena e Reggio Emilia, Italia

http://www.imagelab.ing.unimore.it

The challenge of tracking Social Groups in Crowd

UNIMORE University of Modena and Reggio Emilia

Post-CVPR AC Meeting Workshop on Recent trends in computer vision University of Maryland , Feb 2014

UN

IMO

RE

Un

iversity o

f Mo

den

a and

Reggio

Emilia

ImageLab: current projects..

Computer Vision Pattern Recognition and machine learning

Multimedia

Surveillance (NATO, EU projects, collaboration with companies, SMART TOURISM project Emilia R. Region with S.Calderara)

Ego-Vision (collaboration with ETHZ with G.Serra)

Animal behavior (collaboration

with Italian Health Ministry with S.Calderara)

Document analysis (collaboration with Treccani Italian Enciclopedy and Miniature Libs with C.Grana)

Medical imaging (EU projects in dermatology , C.Grana)

Sensing floors (collaboration

with FLORIM spa with R.Vezzani)

Web image retrieval for cultural heritage (ITALIAN PON

Project EU-FESR DICET)

Natural interaction for children ( Cluster Project smart city CITYEDU)

Current PROJECTS

UN

IMO

RE

Un

iversity o

f Mo

den

a and

Reggio

Emilia

In Smart City Projects..

Goal Understand what the people want/their intentions in the city while they are:

ALONE IN GROUPS IN THE CROWD

TOURISTIC TOURS – CULTURE ENTERTAINMENT – CHILDREN IN SCHOOLS

From surveillance to human behavior analysis….

UN

IMO

RE

Un

iversity o

f Mo

den

a and

Reggio

Emilia

Reasoning about CROWDS

What is a CROWD? a large number of persons gathered closely together

What does LARGE mean?

We are working on crowds where single person and

groups can be recognized.

UN

IMO

RE

Un

iversity o

f Mo

den

a and

Reggio

Emilia

Before understanding groups..

Many challenges:

What we are doing at Imagelab:

• Detecting single people

• Tracking single people

• Tracking multiple people

• Working on trajectories (or tracklets)

• Recognizing (socially consistent) groups in crowd

• By shape classification

• By trajectory analysis

ENVI-VISION EGO-VISION

UN

IMO

RE

Un

iversity o

f Mo

den

a and

Reggio

Emilia

Detecting people ..

Pedestrian detectors a long story… • Detectors: Dalal, Triggs CVPR05, Felzenszwalb, CVPR08, Gavrila et al PAMI09……….

• Benchmarks: Dollar et al CVPR09

• Search modes : Lampert et al CVPR08

• Detection in crowd: Ge Collins PETS09, Li et al. CVPR13

• Detection and tracking in crowd: Rodriguez et al. ICCV11

• Survey: Dollar et al TPAMI11…

Improving speed and accuracy “Multi-Stage Particle Windows for

Fast and Accurate Object Detection”

[Gualdi, Prati, Cucchiara TPAMI12]

form sliding windows to particle windows

search for people (and other targets)

UN

IMO

RE

Un

iversity o

f Mo

den

a and

Reggio

Emilia

..and tracking single (people) target

Is tracking a solved problem? Another long story from L.Davis W4 CVPR98 ICIAP99…..

We tried to answer this questions in an “experimental evaluation”

Even in case of single target tracking*

- a very large dataset

of 14 categories of challenges

- a large set of performance measures

- a large experimentation

(with code available over 3 clusters in 3 labs)

* D.Chu, A.Smeulders, S.Calderara, R.Cucchiara, A. Dehghan, M.Shah Visual Tracking: an Experimental Survey [TPAMI 2013]

MOTA; OTA; Deviaton…. F-Measure SURVIVAL CURVES..

19 trackers BASELINES STATE OF THE ART

UN

IMO

RE

Un

iversity o

f Mo

den

a and

Reggio

Emilia

19 Trackers

A. Tracking by Matching

B. Tracking by Matching with extended model (ST memory)

C. Tracking by Matching with constraints

D. Tracking by Discriminant Classification

E. Tracking by discriminant Classification

with constraints

• [NCC] Normalized Cross-Correlation K. Briechle and U. Hanebeck, SPIE 2001

• [KLT] Lucas-Kanade Tracker S. Baker and I. Matthews, IJCV2004

• [KAT] Kalman Appearance Tracker H. Nguyen and A. Smeulders, TPAMI 2004

• [FRT] Fragments-based Robust Tracking A. Adam, E. Rivlin, and I. Shimshoni, CVPR2006

[MST] Mean Shift Tracking D. Comaniciu, V. Ramesh, and P. Meer, CVPR2000

• [LOT] Locally Orderless Tracking S. Oron, A. Bar-Hillel, D. Levi, S. Avidan, CVPR2012

• [IVT] Incremental Visual Tracking D. Ross, J. Lim, and R.S.Lin, IJCV2008

• [TAG] Tracking on the Affine Group J. Kwon and F.C. Park, CVPR2009

• [TST] Tracking by Sampling Trackers J. Kwon, K.M. Lee, ICCV 2011

• [TMC] Tracking by Monte Carlo sampling J. Kwon, K.M. Lee,CVPR 2009

• [ACT] Adaptive Coupled-layer Tracking L. Cehovin, M. Kristan, A. Leonardis, ICCV2011

• [L1T] L1-minimization Tracker X. Mei and H. Ling, ICCV2009

• [L1O] L1 Tracker with Occlusion detection X. Mei, H. Ling, Y. Wu, E. Blasch, L. Bai, CVPR2011

• [MIT] Multiple Instance learning Tracking B. Babenko, M.H. Yang, and S. Belongie, CVPR2009

• [TLD] Tracking, Learning and Detection Z. Kalal, J. Matas, and K. Mikolajczyk, CVPR2010

• [FBT] Foreground-Background Tracker H. Nguyen and A. Smeulders, 2006, IJCV2010

• [HBT] Hough-Based Tracking M. Godec, P.M. Roth, H.Bischof, ICCV2011

[SPT] Super Pixel tracking S. Wang, H. Lu, F. Yang, M.H. Yang, ICCV2011

• [STR] STRuck S. Hare, A. Saffari, P. Torr, ICCV2011

UN

IMO

RE

Un

iversity o

f Mo

den

a and

Reggio

Emilia

14 tracking challenges in 313 videos

01-LIGHT

02-SURFACECOVER

03-SPECULARITY

04-TRANSPARENCY

05-SHAPE

06-MOTIONSMOOTHNESS

07-MOTIONCOHERENCE

08-CLUTTER

09-CONFUSION

10-LOWCONTRAST

11-OCCLUSION

12-MOVINGCAMERA

13-ZOOMINGCAMERA

14-LONGDURATION

UN

IMO

RE

Un

iversity o

f Mo

den

a and

Reggio

Emilia

http://www.alov300.org or http://imagelab.ing.unimo.it/dsm

The dataset: an example

email to simone.calderara@unimore.it

UN

IMO

RE

Un

iversity o

f Mo

den

a and

Reggio

Emilia

A comprehensive view Survival curve

A B C D E

[NCC]

[STR]

[L1O]

[TST]

[TLD]

[FBT]

The upper bound, taking the best of all trackers at each frame 10%

The lower bound, what all trackers can do 7%

About the 30%, correctly tracked only

UN

IMO

RE

Un

iversity o

f Mo

den

a and

Reggio

Emilia

Confusion challenge: trackers comparison

[FBT][NCC][STR] [TLD][TST] [L1O]

CONFUSION.. CROWD short term tracking

UN

IMO

RE

Un

iversity o

f Mo

den

a and

Reggio

Emilia

Long term challenge: trackers comparison

[FBT][NCC][STR] [TLD][TST] [L1O]

We need more effort Welcome to “Long term tracking workshop” at CVPR2014

UN

IMO

RE

Un

iversity o

f Mo

den

a and

Reggio

Emilia

What we learned?

Many observations…

• In cluttered and confusion scenes, Tracking-by-detection methods that use data association , based on discriminative classifiers seem to be promising…..

• Moving from single target to multiple targets in long term cannot be done with multiple instances of a good single-target tracker

• State of the art papers

• Discrete –continue optimization Andriyenko et al CVPR2012

• Continue energy minimization Milan and Roth PAMI2014

• Generalized minim clique Zamire et al ECCV2012

• K-shortest path optimization Berclaz et al PAMI 2011

What do they all have in common?

They are data association techniques that work on already detected pedestrians

UN

IMO

RE

Un

iversity o

f Mo

den

a and

Reggio

Emilia

Work in progress…

Cognitive Visual Tracking with latent structural svms • From neuroscience : two (connected but different) areas for detection ( people,

faces..) and spatio temporal localization (independently by their shapes) • From perceptual psychology : the “object file” theory (Kahnemann, Treisman, Gibbs 1995)

• We use distance only when is possible • Motion prediction and appearance is a plus when useful

Thus?

1. Split the crowd in influence zones (latent knowledge)

2. Decide whether those zones are ambiguous (also latent)

3. Solve unambiguous associations with distance only

4. Employ different level features in ambiguous cases ( ask for shapes, color.. edges.. motion)

http://imagelab.ing.unimore.it/files2/RitaWashington/video/influencezones_tracking.avi

UN

IMO

RE

Un

iversity o

f Mo

den

a and

Reggio

Emilia

Detection and tracklets

UN

IMO

RE

Un

iversity o

f Mo

den

a and

Reggio

Emilia

Survival curve

[KSP] Multiple Object Tracking using K-Shortest Paths Optimization J. Berclaz, F. Fleuret, E. Türetken and P. Fua, PAMI 2011

With a perfect detector

With a detector with errors

UN

IMO

RE

Un

iversity o

f Mo

den

a and

Reggio

Emilia

Groups of People

If tracking was solved…

If we were given the trajectories of every pedestrian in the scene (more or less). would we be able to discern the presence of groups?

UN

IMO

RE

Un

iversity o

f Mo

den

a and

Reggio

Emilia

Detecting social groups in crowds

Group detection: learn to partition into groups the pedestrians being part of a crowd observing pairwise relations and transitivities.*

Integrating two cues:

* Structured learning for detection of social groups in crowd Solera, Calderara, Cucchiara, AVSS 2013

2. GRANGER CAUSALITY • Intuition: two pedestrian belonging to the

same group will probably influence each other position and direction!2

• The Granger causality test is a statistical hypothesis test for determining whether one time series is useful in forecasting another

1. HALL’S PROXEMICS • Hall’s proxemics theory 1 defines reaction

bubbles around every individual and

• the interaction between pairs of individuals can be classified according to a quantization of their mutual distance

UN

IMO

RE

Un

iversity o

f Mo

den

a and

Reggio

Emilia

Results

Features: Proxemics and Granger causality Structure function: pair-wise correlation clustering Group detection: Structured SVM [groups]

UN

IMO

RE

Un

iversity o

f Mo

den

a and

Reggio

Emilia

Conclusions and Open Problems

• Single target Detection & Tracking

• tracking is (still) an open problem

• computer visionaries are working a lot.. ( also in the weekend )

• Multiple target tracking

• more and more challenging ( more if real-time is required)

• tracking-by-detection

• Cognitive assumptions are useful

• Detection Social groups and interactions

• interesting and growing topic

• Many many many applications

• Social hypotheses Must be considered

.

People/ group Detection

People/ group tracking

People/ group Detection

People/ group tracking

People Detection

People/ group Tracking

Social group Detection

UN

IMO

RE

Un

iversity o

f Mo

den

a and

Reggio

Emilia

Thanks

PEOPLE @ http://imagelab.ing.unimore.it

Rita Cucchiara Giuseppe Serra Marco Manfredi

Costantino Grana Paolo Santinelli Francesco Solera

Roberto Vezzani Martino Lombardi Simone Pistocchi

Simone Calderara Michele Fornaciari Fabio Battilani

Dalia Coppi Patrizia Varini Augusto Pieracci

THANKS!

UN

IMO

RE

Un

iversity o

f Mo

den

a and

Reggio

Emilia