1

Presentation of Vista team(VIsion Spatio-Temporelle et Apprentissage)

Patrick Bouthemy

UR Rennes, IRISA

(http://www.irisa.fr/vista/)

2

INRIAP. Bouthemy, F. Cao (2001), I. Laptev (2005),

P. Pérez (2004),

C. Kervrann (2003, on Inria secondment, Inra)

CNRS J. P. Le Cadre

University of Rennes 1 E. Mémin

J. Yao (scientific collaborator, Maths dept., Irmar)

Secretary :: H. Béchu (Inria, with Temics team)

VISTA-2005 research staff

3

Ph-D students (11)

Post-docs (2) I. Laptev (until August 2005),

Venkatesh Babu (until March 2005)

Temporary technical staff (3) N. Gengembre (FT-RD contract), P. Heas (IST-FET Fluid),

B. Fauvet (RIAM Feria project, until June 2005)

VISTA-2005 research staff – continued

G. Piriou, T. Bréhard, T. Veit (tbd, Dec. 2005)

V. Auvray, A. Cuzol, J. Boulanger (3rd year)

A. Bugeau, N. Papadakis (2nd year)

T. Crivelli, C. Simonin, A. Hervieu (1st year)

4

Research topics (1) Overall objective: Analyzing dynamic scenes or physical

phenomena from image sequences (video, IR, satellite images, video-microscopy)

Involved topics:Restoring image sequencesDetecting, segmenting, tracking moving entitiesMeasuring and representing image motionModeling, learning and recognizing spatio-temporal contents

Problem formulation based on visual (2D) motion models and measurements

5

Research topics (2)

Applied maths foundations: mostly statistical approaches for image analysis

Probabilistic models (Markov models, probabilistic mixed-state models)

Bayesian inference, robust estimation, non-parametric estimation

A contrario decision methodsParticle filteringRobust classification

6

Application domains

Content-aware video applications (multimedia and TV, sports videos, enriched video, video summarization, video indexing, video re-purposing) main partners: INA, Thomson, FT-RD projects: IST Lava (XRCE, Lear team), RNTL Domus Videum (Atlas, Metiss

and Temics teams), RIAM Feria (Texmex team), IST-NoE Muscle (Ariana, Imedia and Texmex teams), FT-RD contract, ACI Behaviour (UTC, PSA)

Experimental fluid mechanics and Meteorological imagery partners: Cemagref, LMD projects: Eumetsat, IST-FET Fluid, ACI Assimage (Clime and Idopt teams)

Biological imagery (video-microscopy) partners: INRA, Curie Institute, Univ. Rennes 1 (Dept of Biology) projects: ACI-IMPBio MoDynCell5D, AC DRAB

7

Main evolutions 2001- 2005

Important changes in the team composition Creation of Texmex team (2002, P. Gros, 2 people from Vista, multimedia

indexing, as announced in 2001) Creation of Lagadic team (2004, F. Chaumette, E. Marchand, F. Spindler,

11 people all from Vista, as announced in 2001) Creation of Visages team (2004, C. Barillot, P. Hellier, S. Prima, 6 people

from Vista, medical imaging) Arrival of 4 research scientists over 2001-2005 (3 Inria, 1 Inra)

Given the present Vista team composition and the corresponding re-focusing (in 2004) on dynamic scene analysis: Achievement of (or at least investigation of) all the objectives listed in

2001 (apart from a minor point, NDC of structures under strain) Addition of two new topics (a contrario motion analysis, video-microscopy)

and reinforcement of a third one (video analysis and tracking).

8

Main contributions (1) Adaptive non-parametric estimation for image sequence denoising

Efficient method based on locally adaptive kernel regression while

preserving space-time discontinuities without motion estimation Data-driven adaptive estimation window based on bias-variance trade-off

criterion Point-wise and patch-based versions

Experimental comparative evaluation on usual video image sequences: our method outperforms other recent methods (joint Kalman-Wiener, wavelet-based, and PDE-based methods)

Left: original sequence; middle: noisy sequence; right: denoised sequence with our method

9

Main contributions (1bis) Application to video-microscopy

Goal: detection and tracking of moving intra-cell protein vesicles to model and interprete their dynamics for biological studies

First issue addressed: 3D image sequence denoising

Positioning (national and international context): Pasteur Institute (J.-C. Olivo-Marin), ENS Paris, EPFL (M. Unser), Heidelberg Univ.Image sequence provided Denoised sequence

by Curie Institute, with our methodstudy of Rab6 protein

One 3D imageof the sequence:noisy (up) anddenoised (bottom)

10

Main contributions (2) A contrario methods for motion detection and matching

Decision is performed a contrario: large deviation (very small probability, NFA computation) to a random background model (independence assumption), no prior required on the searched entities, importance of the measurement choice.

General, simple and parameter-free (automatic threshold setting) method while supplying confidence values.

Pos.: ENS Cachan (J.-M. Morel, a contrario framework for static image analysis, collaboration on shape matching and recognition), Ceremade (F. Dibos), UPF Barcelona (V. Caselles)

11

Main contributions (2bis)

Addressed problems A contrario instantaneous detection of moving regions A contrario coherent motion detection (clustering in an appropriate

space, initialization for tracking)

A contrario image comparison (global and local criteria) and retrieval in video streams

12

Main contributions (3) Mixed-state probabilistic models for motion modeling and recognition

A unique representation for information which can be both discrete (symbolic) and continuous. No supplementary (hidden) label field requiring an inference stage.

Definition of specific mixture distributions (Dirac + exponential family) to model (global or local) occurrence statistics of 2D motion measurements for motion classification, supervised motion recognition and event detection (in

sports videos). Recognition of serve shots

Definition of mixed-state auto-models (extension of Besag’s framework to multi-parameter cases) and first application to the modeling of temporal (motion) textures.

Pos.: UCF (M. Shah), Weizmann Institute (M. Irani), Microsoft Research (H.-J. Zhang), Univ. Rochester (M. Tekalp), UCLA (S. Soatto, linear models for dynamic textures)

(false detection)

13

Main contributions (4)

Fluid motion analysis

Estimation of dense velocity fields for fluid flowsSpecific data model (continuity equation) and second-order div-curl regularization

Parsimonious decomposition on adapted basis functions (vortex and source particles)Estimation of the motion of multiple layers (exploiting a prior image segmentation)Intensive evaluation on typical fluid flow experiments (in collaboration with Cemagref)

Pos.: Correlation-based methods in applied meteorology (wind field computation), PIV (Particle Image Velocimetry) techniques in experimental fluid mechanics

(Image sequence provided by Onera)

14

Main contributions (4bis) Fluid motion analysis (continued)

Structuration of the fluid flow fieldEstimation of the potential functions associated with the irrotational and solenoidal components of the flow field (Fourier and variational approaches)

Tracking of high-dimensional flow structures (e.g., motion fields)Stochastic filtering relying on the vorticity-velocity form of Navier-Stokes equation and then Itô diffusion processDeterministic filtering exploiting a variational data assimilation technique

Tracking of a cycloneby stochastic filtering

16 vortex particles1000 filtering particles

15

Main contributions (5) Non-linear tracking methods

Robust tracking in low-dimensional state-space and complex likelihoodsMulti-object tracking with particle filters (novel SMC techniques, data-target association issue) New conditional formulation of classical filters with state equation estimated from the image data (point and planar structure trackers) Tracking with auxiliary variables (handling of visibility, occlusion events)

Left: our method (item 2), all the points are correctly tracked,Right: STK tracker (several points are lost) Tracking with auxiliary variables

of visibility

16

Main contributions (5bis) Non-linear tracking methods (continued)

High-dimensional state spaces and detailed dynamicsSee previous slide on fluid motion analysis

Target tracking for partially observed non-linear systemsInitialization of particle filtering based on a hierarchical approachDerivation of the PCRB closed-form (Posterior Cramer-Rao Bound) for performance analysis and sensor managementDesign of decentralized particle filters

Pos.: Georgia Tech (F. Delleart), EPFL (P. Fua), ETHZ (L. van Gool), Brown University (M. Black), Siemens Princeton (D. Comaniciu), Microsoft Research (A. Blake), DSTO Adelaide (N. Gordon), Qinetiq UK (S. Maskell)

Distributed target tracking with two observers (in blue) True trajectory in green, estimated trajectories obtained by each observer in red. Confidence areas for each observer in red are large. Distributed target tracking in black. Confidence area in black is very small.

17

Some highlights (1)

Academic dissemination and visibility 16 Ph-D theses, 62 journal papers (2001-2005), 2 books, 2 best paper awards Co-general chair of ICCV’07, organization of CBMI’03 (of Miccai’04, Visages

team), PC co-chair of RFIA’06, Associate editors IEEE-IP, Area editor JAIF.

Involvements in industrial partnership, applications and transfer Consortium (GIS Aeternum Multimedia) formed by Thomson, INA and Irisa

(Metiss, Texmex and Vista teams)

Cross-modal video summarization tool validated on football and rugby TV programs (Thomson)

FT-RD contract on team sports video processing (Rugby world cup’2007)

Head of European FLUID project (LaVision PIV company); transfer of estimation tool of dense fluid flow fields to meteorological domain (Eumetsat) and experimental fluid mechanics one (Cemagref, Onera)

18

Some highlights (2) International collaborations

Associate Inria team FIM (University of Buenos-Aires, on fluid motion analysis and on dynamic texture analysis)

Univ. of Mannheim (C. Schnörr), Univ.of Cambridge (J. Vermaak), Univ. of Las Palmas (L. Alvarez), UCSD (S. Belongie), European NoE Muscle

Software Motion-2D (source code for estimating parametric motion models, publically

available under QPL): about 550 downloads (explicitly identified) since August 2003 (http://www.irisa.fr/vista/Motion2D/)

Tracking application (on-going development)

Platform Multimedia platform (with Texmex and Metiss teams, collaboration with

INA, starting 2005): capture of TV programs, content server, metadata server, multimedia processing, content-based multimedia indexing, experimental evaluation,…

19

Perspectives (1)

Video analysis and understanding

Analysis of motion textures with mixed-state models (segmentation, classification)

Modeling and recognition of dynamic video contents from bags of trajectories

Video alignment (dynamic multi-view constraints) for content recognition and retrieval

Combined motion and object categorization and recognition Learning issues: semi-supervised learning with tracking for video

contents; dynamic kernel-based methods Specific focus on sports TV programs (cross-modal analysis, self-

structuration, video summarization)

20

Perspectives (2)

Tracking problems

Visual tracking with no prior (on-line learning and updating of appearance model, cluttered background, combination of complementary representations, conditional Markov models)

Camera network (partially observed dynamic system, low-level measurements, polyedral separation methods, trajectory reconstruction, situation analysis for groups of moving entities)

High dimensionality in visual tracking problems (variational assimilation techniques and non-linear Bayesian filters)

Trajectory optimization for terrain-aided navigation (collaboration with Onera)

21

Perspectives (3) Image sequence analysis for environmental, fluid mechanics

and biological studies

Further incorporation of physical laws 3D motion estimation from 3D volume data or from multi-view 2D

data (in experimental fluid mechanics and meteorological applications) New involvement in oceanography applications

Continuation of our work in video-microscopy on the detection and tracking of small moving intra-cell entities

Interpretation of analysed dynamic contents in relation with biological studies (partnership with biologists)

Envisaged proposition (in 2-3 years) of a new joint Inria-Inra team devoted to biological image sequence analysis and related applications