View
214
Download
0
Category
Preview:
Citation preview
PULSAR Perception Understanding
Learning Systems for Activity Recognition
Theme: Cognitive Systems Cog C Multimedia data: interpretation and man-machine interaction
Multidisciplinary team:Computer vision, artificial intelligence, software engineering
P U L S A R
P U L S A R
September 2007
2
5 Research Scientists: François Bremond (CR1 Inria, HDR)
Guillaume Charpiat (CR2 Inria, 15 December 07)
Sabine Moisan (CR1 Inria, HDR)
Annie Ressouche (CR1 Inria)
(team leader) Monique Thonnat (DR1 Inria, HDR)
1 External Collaborator: Jean-Paul Rigault (Prof. UNSA)
1 Post-doc: Sundaram Suresh (PhD Bangalore, ERCIM)
5 Temporary Engineers: B. Boulay (PhD) , E. Corvee (PhD)
R. Ma (PhD) , L. Patino (PhD) , V. Valentin
8 PhD Students: B. Binh, N. Kayati, L. Le Thi, M.B. Kaaniche,
V. Martin, A.T. Nghiem, N. Zouba, M. Zuniga
1 External visitor: Tomi Raty (VTT Finland)
Team presentation
P U L S A R
September 2007
3
Objective: Cognitive Systems for Activity Recognition
Activity recognition: Real-time Semantic Interpretation of Dynamic
Scenes
Dynamic scenes: Several interacting human beings, animals or vehicles
Long term activities (hours or days)
Large scale activities in the physical world (located in large space)
Observed by a network of video cameras and sensors
Real-time Semantic interpretation: Real-time analysis of sensor output
Semantic interpretation with a priori knowledge of interesting behaviors
PULSAR
P U L S A R
September 2007
4
Objective: Cognitive Systems for Activity Recognition
Cognitive systems: perception, understanding and learning
systems Physical object recognition
Activity understanding and learning
System design and evaluation
Two complementary research directions: Scene Understanding for Activity Recognition
Activity Recognition Systems
PULSAR Scientific objectives:
P U L S A R
September 2007
5
Two application domains: Safety/security (e.g. airport monitoring)
Healthcare (e.g. assistance to the elderly)
PULSAR target applications
P U L S A R
September 2007
6
Cognitive Systems for Activity Recognition
Airport Apron Monitoring
Outdoor scenes with complex interactions between humans, ground vehicles, and aircrafts
Aircraft preparation: optional tasks, independent tasks, temporal constraints
P U L S A R
September 2007
7
Cognitive Systems for Activity Recognition
Monitoring Daily Living Activities of Elderly
Goal: Increase independence and quality of life: Enable people to live at home Delay entrance in nursing home Relieve family members and caregivers
Approach: Detecting changes in behavior (missing activities, disorder, interruptions, repetitions, inactivity) Calculate the degree of frailty of elderly peopleExample of normal activity:
Meal preparation (in kitchen) (11h– 12h)Eat (in dinning room) (12h -12h30) Resting, TV watching, (in living room) (13h– 16h) …
P U L S A R
September 2007
8
Gerhome laboratory (CSTB,PULSAR) http://gerhome.cstb.fr
Water sensor
Contact sensors to detect “open/close”
Presence sensor
P U L S A R
September 2007
9
Orion contributions 4D semantic approach to Video Understanding Program supervision approach to Software Reuse VSIP platform for real-time video understanding
Keeneo start-up LAMA platform for knowledge-based system design
From ORION to PULSAR
P U L S A R
September 2007
10
1) New Research Axis:
Software architecture for activity recognition
2) New Application Domain:
Healthcare (e.g. assistance to the elderly)
3) New Research Axis:
Machine learning for cognitive systems (mixing perception,
understanding and learning)
4) New Data Types:
Video enriched with other sensors (e.g. contact sensors, ….)
From ORION to PULSAR
P U L S A R
September 2007
11
Perception for Activity Recognition (F Bremond, G Charpiat, M
Thonnat)
Goal: to extract rich physical object description
Difficulty: to obtain real-time performances and robust detections in dynamic and complex situations
Approach: Perception methods for shape, gesture and trajectory description
of multiple objects Multimodal data fusion from large sensor networks sharing same
3D referential Formalization of the conditions of use of the perception methods
PULSAR research directions
P U L S A R
September 2007
12
Understanding for Activity Recognition (M Thonnat F Bremond S
Moisan) Goal: physical object activity recognition based on a priori models Difficulty: vague end-user specifications and numerous
observations conditions
Approach: Perceptual event ontology interfacing the perception and the
human operator levels Friendly activity model formalisms based on this ontology Real-time activity recognition algorithms handling perceptual
features uncertainty and activity model complexity
PULSAR research directions
P U L S A R
September 2007
13
Learning for Activity Recognition (F Bremond, G Charpiat, M
Thonnat) Goal: learning to decrease the effort needed for building activity
models Difficulty: to get meaningful positive and negative samples Approach:
Automatic perception method selection by performance evaluation and ground truth
Dynamic parameter setting based on context clustering and parameter value optimization
Learning perceptual event concept detectors Learning the mapping between basic event concepts and
activity models Learning complex activity models from frequent event
patterns
PULSAR research directions
P U L S A R
September 2007
14
Activity Recognition Systems (S Moisan, A Ressouche, J-P
Rigault) Goal: provide new techniques for easy design of effective and
efficient activity recognition systems Difficulty: reusability vs. efficiency
From VSIP library and LAMA platform to AR platform
Approach: Activity Models: models, languages and tools for all AR tasks Platform Architecture: design a platform with real time
response, parallel and distributed capabilities System Safeness: adapt state of the art verification &
validation techniques for AR system design
PULSAR research directions
P U L S A R
September 2007
15
PULSAR: Scene Understanding for Activity
Recognition Perception: multi-sensor fusion, interest points and mobile regions,
shape statistics Understanding: uncertainty, 4D coherence, ontology for activity
recognition Learning: parameter setting, event detector, video mining
PULSAR: Activity Recognition SystemsFrom LAMA platform to AR platform: Model extensions: modeling time and scenarios Architecture: real time response, parallelization, distribution User-friendliness and safeness of use: theory and tools for
component framework, scalability of verification methods
Objectives for the next period
P U L S A R
September 2007
16
Person recognition
Multimodal Fusion for Monitoring Daily Living Activities of Elderly
Meal preparation
activity
Resting in living
room activity
Person recognition
Multimodal recognition
3D Posture recognition
P U L S A R
September 2007
17
Multimodal Fusion for Monitoring Daily Living Activities of Elderly
Resting in living room activity
Person recognition 3D Posture recognition
P U L S A R
September 2007
18
Person recognition
Multimodal Fusion for Monitoring Daily Living Activities of Elderly
Meal preparation activity
Multimodal recognition
P U L S A R
September 2007
19
Understanding and Learning for Airport Apron Monitoring
European project AVITRACK (2004-2006) predefined activities
European project COFRIEND (2008-2010) activity learning, dynamic configurations
Activity Recognition Platform Architecture
Component level
Task level
Application level
Understanding components
Learning components
Perception components
Object recognition and tracking
Scenario recognition
Airport monitoring
Program supervision
Vandalism detection
Elderly monitoring
Configuration and deployment tools
Communication and interaction facilities
Ontology management
Parsergeneration
Usage support tools
Componentassembly
VerificationSimulation& testing
P U L S A R
September 2007
22
Video Data MiningObjective: Knowledge Extraction for video activity monitoring
with unsupervised learning techniques.Methods: Trajectory characterization through clustering (SOM)
and behaviour analysis of objects with relational analysis1.
2
2
2
wk pp
w
kiwkk
ek
mtkmm
clustersnbKKk _:;...1
m1
m2
mk
mK
22_:
_:;...1
kiwi
thi
mtmtneuronwinningw
estrajectorinbnni
trajectoryit
Self Organizing Maps (SOM) Relational AnalysisAnalysis of the similarity between two individuals i,i’ given a variable :
jiic '
jV
. .
.
.
jV
1O 'iO nO
1O
iOjV
iic '
nO
1 BENHADDA H., MARCOTORCHINO F., Introduction à la similarité régularisée en analyse relationnelle, revue de statistique, Vol. 46, N°1, pp. 45-69, 1998
P U L S A R
September 2007
23
Video Data Mining Results
2052 trajectories
Step 1:Trajectoryclustering (SOM)
Trajectory Cluster 1: Walk from north doors to vending machines
Step 2: Behaviour Relational Analysis
Behavior Cluster 19: Individuals and not Groups buy a ticket at the entrance
Trajectory Cluster 9: Walk from north gates to south exit.
P U L S A R
September 2007
24
Scenario for Meal preparation
Composite Event (Use_microwave,
Physical Objects ( (p: Person), (Microwave: Equipment), (Kitchen: Zone))
Components ((p_inz: PrimitiveState inside_zone (p, Kitchen))
(open_mw: PrimitiveEvent Open_Microwave (Microwave))
(close_mw: PrimitiveEvent Close_Microwave (Microwave)) )
Constraints ((open_mw during p_inz )
(open_mw->StartTime + 10s < close_mw->StartTime) ))
Multimodal Fusion for Monitoring Daily Living Activities of Elderly
Detected by video camera Detected by contact sensor
Recommended