Upload
gerda
View
35
Download
0
Tags:
Embed Size (px)
DESCRIPTION
Fusion of Multiple Cues from Color and Depth Domains using Occlusion Aware Bayesian Tracker. Kourosh MESHGI Shin- ichi MAEDA Shigeyuki OBA Shin ISHII 18 MAR 2014. Integrated System Biology Lab (Ishii Lab) Graduate School of Informatics Kyoto University [email protected] - PowerPoint PPT Presentation
Citation preview
Fusion of Multiple Cues from Color and Depth Domains using
Occlusion Aware Bayesian Tracker
Kourosh MESHGIShin-ichi MAEDAShigeyuki OBAShin ISHII
18 MAR 2014
Integrated System Biology Lab(Ishii Lab)Graduate School of InformaticsKyoto [email protected]
IEICE NC Tamagawa’14
TRACKING APPLICATIONS
K O U R O S H M E S H G I – I S H I I L A B - D E C 2 0 1 3 - S L I D E 2
MAIN APPLICATIONS
Surveillance Public Entertainment
Robotics Video Indexing
Action Recog.
TRACKING CHALLENGES
K O U R O S H M E S H G I – I S H I I L A B - D E C 2 0 1 3 - S L I D E 3
MAIN CHALLENGES
Varying ScaleClutterNon-Rigid
OcclusionIlluminationAbrupt Motion
[Mihaylova et al., 07] RGB Color + Texture + Motion + Edge, Two PF
[Spinello & Arras,11] HOG on RGB + HOG on Depth, SVM Classification
[Shotton et al, 11] Skeleton from Depth, Random Forrest
[Choi et al, 11] Ensemble of Detectors (upper body, face, skin, shape from depth, motion from depth), RJ-MCMC
LITERATURE REVIW(Channel Fusion & Occlusion)
[Song et al., 13] 2.5D Shape + Motion + HOG on Color and Depth, Occlusion Indicator, SVM
K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 5
OBSERVATION MODELFrame: t
Observation
, ,{ , }t rgb t d tI I I
Image Patch
K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 6
STATE REPRESENTATIONFrame: t
State
{ , , , }t t t t tB x y w h{ }t tX B
( ; )t t tY g I B
w
h
(x,y)
K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 7
FEATURES
Feature Set1{ ,..., }nF f f
Color
Shape Edge
Texture
K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 8
TEMPLATE INIT.Frame: 1
Template1 1,1 ,1{ ,..., }n
f1 fj fn
1 ,1 1{ }ni i
1 1{ ( )}if Y
… …
K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 9
PARTICLES INITIALIZATIONFrame: 1
Particles, ,{ }k t k tX B1,2, ,k N
Initialized Overlapped
K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 1 0
MOTION MODELFrame: t
Motion Model, , ,k t k t k tB B
→ t + 1
K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 1 1
FEATURE EXTRACTIONFrame: t + 1
Feature Vectors , 1( )i k tf Y
f1 f2 fn
X1,t+1
X2,t+1
XN,t+1
…
…
K O U R O S H M E S H G I – I S H I I L A B - M A R 2 0 1 4 - S L I D E 1 2
FEATURE FUSIONFrame: t
Probability of Observation( | )t tp Y X ,1
( | , )ni t t i ti
p Y B ( | , )t t tp Y B 1, 2, ,( | , , , , )t t t t n tp Y B 1 1, ,( | , ) ( | , )t t t n t t n tp Y B p Y B ,1
( ),ni i i t i ti
p D f Y ,1
exp ( ),ni i i t i tiD f Y
,1
exp ( ),ni i i t i tiD f Y
Each Feature(.)if(.)iD
i Indepen
dence
Assumptio
n
!
K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 1 3
PROB. CALCULATIONFrame: t + 1
Particles
Brighter = More Probable
,k tp
K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 1 4
TARGET ESTIMATIONFrame: t + 1
Expectation 1 ,
ˆt k tB B E
K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 1 5
MODEL UPDATEFrame: t + 1
New Model
Model Update
1ˆ ˆ( ; )t t tY g I B
1ˆ ˆ( )t i tf Y
, 1 , 1
,
ˆ
(1 )i t i i t
i i t
K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 1 6
RESAMPLINGFrame: t + 1
Proportional to Probability
1( | )t tp X X1
2
345
67
K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 1 7
APPEARANCE CHANGES
Same Color ObjectsBackground ClutterIllumination ChangeShadows, Shades
Use Depth!
K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 1 8
MODEL DRIFT PROBLEM
Templates Corrupted! t
Handle Occlusion!
PERSISTENT OCCLUSION
Particles Converge to Local Optima / Remains The Same Region
K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 1 9
Advanced Motion Models(not always feasible)
Restart Tracking(slow occlusion recovery)
Expand Search Area!
OCCLUSIONdo not address occlusion explicitly
maintain a large set of hypotheses
computationally expensive
direct occlusion detection robust against partial & temp occ. persistent occ. hinder tracking
GENERATIVE MODELS DISCRIMINATIVE MODELS
Dynamic Occlusion: Pixels of other object close to camera
Scene Occlusion: Still objects are closer to camera than the target object
Apparent Occlusion: Due to shape change, silhouette motion, shadows, self-occ
UPDATE MODEL FOR TARGET TYPE OF OCCLUSION IS IMPORTANT KEEP MEMORY VS. KEEP FOCUS ON THE TARGET
Combine
Them!
DYNAMICS…
* The Search is not Directed* Neither of the Channels have Useful Information* Particles Should Scatter Away from Last Known Position
K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 2 1
Occlusion!
Occlusion Flag (for each particle)
Observation Model
No-Occlusion Particles Same as Before
Occlusion-Flagged Particles Uniform Distribution
OCCLUSION AWAREPARTICLE FILTER FRAMEWORK
( | ) ( | , , )t t t t t tp Y X p Y B Z ( | ) (1 ) ( | , 0, ) ( | , 1, )t t t t t t t t t t t tp Y X Z p Y B Z Z p Y B Z
,k tZ
( | , 1, ) 1t t t tp Y B Z
,1( | , 0, ) exp ( ),n
t t t t i i i t i tip Y B Z D f Y
K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 2 2
Probability of Occlusion for the Next Box
Modified Dynamics Model of Particle
PARTICLE FILTER DYNAMICS
1 , , , , ,1 1ˆ ( ) ( ) ( | )N N
t t i t i t i t i t i ti iZ Z p Z Z p X Y Z
E
1 1 1 1 1( | ) ( , | , ) ( | ) ( | )t t t t t t t t t tp X X p B Z B Z p B B p Z Z
K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 2 3
Model Update
Separately for each Feature
UPDATE RULE
11
1 1
ˆ( ) ,( )
ˆ ˆ( ) (1 ) ( ) ,t t occ
t
t t t occ
f Zf
f Y f Z
K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 2 4
OA-PF DYNAMICS
K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 2 5
Occlusion!
OA-PF DYNAMICS
K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 2 6
Occlusion!
GOTCHA!
OA-PF DYNAMICS
K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 2 7
Quick Occlusion Recovery Low CPE
No Template Corruption
No Attraction to other Object/ Background
CO
LO
R
(HO
C)
TE
XT
UR
E
(LB
P)
ED
GE
(L
OG
)
DE
PTH
(H
OD
)
3D SH
APE
(PC
L Σ)FEATURES
K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 2 8
Princeton Tracking Dataset
DATASET( )
5 Validation Video with Ground Truth95 Evaluation Video
K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 2 9
EXPERIMENT
K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 3 0
A. Edge PF
B. Edge + Color PF
C. Edge + Color + Depth PF
D. Edge + Color + Depth + Texture PF
E. Edge + Color + Depth + Texture + 3D Shape PF
F. Occlusion Aware PF
DEMONSTRATION(Yellow Dashed Line is Ground Truth)
PASCAL VOC
CRITERIA I
1
1
*1
* *1 1 1
*1 1
*1 1
ˆ
ˆ ˆ, 0ˆ1 , 1ˆ1 ,
t
t
t
t t t
t t t
t t
B B
B B Z Z
S Z Z
Z Z
0 1ott oS t AUC
toSu
cces
sOverlap Threshold
0
1
1
Area Under Curve
K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 3 2
RESULTS
K O U R O S H M E S H G I – I S H I I L A B - M A R 2 0 1 4 - S L I D E 3 3
Success Plot A D
B EC F
1
1Overlap Threshold
Succ
ess
Rat
e
Mean Central Point Error: Localization Success
Mean Scale Adaption Error
CRITERIA II
* 2 * 21
ˆˆ( ) ( )Tt t t tt
w w h hSAE
T
* 2 * 21
ˆ ˆ( ) ( )Tt t t tt
x x y yCPE
T
K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 3 4
ˆˆ ˆ ˆ ˆ{ , , , }t t t t tB x y w h * * * * *{ , , , }t t t t t
B x y w h
Estimated Ground Truth
RESULTS
K O U R O S H M E S H G I – I S H I I L A B - M A R 2 0 1 4 - S L I D E 3 5
Center Positioning ErrorA D
B EC F
100
50Frames
CPE
(pix
els)
RESULTS
K O U R O S H M E S H G I – I S H I I L A B - M A R 2 0 1 4 - S L I D E 3 6
Scale Adaptation Error
K O U R O S H M E S H G I – I S H I I L A B - M A R 2 0 1 4 - S L I D E 3 6
140
SAE
(pix
els)
50Frames
A D
B EC F
RESULTS
K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 3 7
Tracker AUC CPE SAE
A (edg)15.7
2192.27
41.62
B (edg+hoc)29.8
893.4
746.7
1
C (edg+hoc+hod)46.7
434.6
240.8
8D (edg+hoc+hod+tex)
48.49
30.18
46.27
E (edg+hoc+hod+tex+shp)
58.03
23.84
29.62
F (all + occlusion handling)
63.58
17.46
25.07
FUTURE WORKS
K O U R O S H M E S H G I – I S H I I L A B – M A R 2 0 1 4 – S L I D E 3 8
More Resilient Features + Scale
Adaptation
Active Occlusion Handling
Measure the Confidence of
each Data Channel
Adaptive Model Update
QUESTIONS?Thank you for your time…
Image Credit: http://www.engg.uaeu.ac.ae/