Upload
others
View
2
Download
0
Embed Size (px)
Citation preview
Learning to Track: Online Multi-Object Tracking by Decision Making
Yu Xiang1,2, Alexandre Alahi1, and Silvio Savarese1
1Stanford University, 2University of Michigan
ICCV 2015
1
Multi-Object Tracking
2
Autonomous driving
Visual surveillance
Sport Analysis
Robot navigation
Batch Mode vs. Online Mode
• Batch Mode
• Online Mode
t-2 ttime axis
t-1 t+1 t+2
t-2 ttime axis
t-1 t+1 t+2
3
Tracking by Detection
4
Data Association
Tracks at time t-1 Detections at time t
time axis
?
5
Challenges
Noisy detection: false alarms and missing detections6
Challenges
Occlusion
7
Similarity Function for Data Association
Tracks at time t-1 Detections at time ttime axis
0.2
0.8
0.3
0.1
8
• Zhang et al., CVPR’08• Berclaz et al., TPAMI’11• Breitenstein et al., TPAMI’11• Pirsiavash et al., CVPR’11• Butt & Collins, CVPR’13• Milan et al., TPAMI’14Etc.
Simple Powerfulsimilarity measure optimization+Ours
Learning to Track
𝜙1( ),Similarity 𝑤1= + ⋯ 𝜙𝑛( ),𝑤𝑛+
Different features/cues between targets and detections
Weights to combine different cues(to be learned)
9
• Appearance• Location• MotionEtc.
Offline-learning vs. Online-learning
10
Offline-learning vs. Online-learning
Offline-learning
Online-learning
Training time Before Tracking
DuringTracking
With supervision
Use history of the target
11
…
…
• Li et al., CVPR’09• Kim et al., ACCV’12Etc.
Offline-learning vs. Online-learning
12
• Song et al., ECCV’08• Kuo et al., CVPR’10• Bae et al., CVPR’14Etc.
Offline-learning
Online-learning
Training time Before Tracking
DuringTracking
With supervision
Use history of the target
The target is tracked
The target is occluded
The target is tracked again
Our Solution: Tracking by Decision Making
13
Inverse Reinforcement Learning
14
tracked lost tracked
Ground truth trajectory
Tracked Lost Tracked
MarkovDecisionProcess(MDP)
Supervision
Comparison between Different Learning Strategies
15
Offline-learning
Online-learning
Ours
Training time Before Tracking
DuringTracking
Before Tracking
With supervision
Use history of the target
Comparison between Different Learning Strategies
16
Offline-learning
Online-learning
Ours
Training time Before Tracking
DuringTracking
Before Tracking
With supervision
Use history of the target
Outline
•Markov Decision Process (MDP) for a Single Target
•Online Multi-Object Tracking with MDPs
• Experiments
•Conclusion
17
Outline
•Markov Decision Process (MDP) for a Single Target
•Online Multi-Object Tracking with MDPs
• Experiments
•Conclusion
18
Active
Tracked
Inactive
Lost
objectdetection
19
Markov Decision Process for a Single Target
Active
Tracked
Inactive
Lost
objectdetection
20
Markov Decision Process for a Single Target
Active
Tracked
Inactive
Lost
objectdetection
21
Markov Decision Process for a Single Target
Active
Tracked
Inactive
objectdetection
Markov Decision Process for a Single Target
22
Markov Decision Process for a Single Target
TLD Tracker. Z. Kalal, K. Mikolajczyk, and J. Matas. Tracking-learning-detection. TPAMI, 34(7):1409–1422, 2012.23
Active
Tracked
Inactive
Lost
objectdetection Single object tracking
Template Tracking in Tracked StatesFrame 50 Frame 51
24
Template Tracking in Tracked StatesFrame 50 Frame 51
25
Template Tracking in Tracked StatesFrame 50 Frame 51
26
Template Tracking in Tracked StatesFrame 50 Frame 51
Tracked
27
Template Tracking in Tracked StatesFrame 50 Frame 57
28
Template Tracking in Tracked StatesFrame 50 Frame 57
29
Template Tracking in Tracked StatesFrame 50 Frame 57
30
Template Tracking in Tracked StatesFrame 50 Frame 57
Tracked
Lost
31
Active
Tracked
Inactive
Lost
objectdetection
Markov Decision Process for a Single Target
If lost for more than T frames
32
Data Association in Lost States
t-2 t
time axis
t-1
tracked lost
?
33
Learning the Similarity Function
𝜙1( ),Similarity 𝑤1= + ⋯ 𝜙𝑛( ),𝑤𝑛+ + 𝑏
34
( ), 1
( ), 2
…( ), M
Hard positive examples
( ), 1
( ), 2
( ), N
…
Hard negative examples
Inverse reinforcement learning: tracking objects in training videos!
Inverse Reinforcement Learning
t-2
1
2
3
4
t
time axis
t-1
tracked lost
35
Ground truth trajectory
Supervision
Inverse Reinforcement Learning
t-2
1
2
3
4
t
time axis
t-1
tracked lost
36
Ground truth trajectory
Supervision
Inverse Reinforcement Learning
t-2
1
2
3
4
t
time axis
t-1
tracked lost
Wrong decision!Update your weights!
37
Ground truth trajectory
Supervision
( ), 1
Negative example
Inverse Reinforcement Learning
t-2
1
2
3
4
t
time axis
t-1
tracked lostWrong decision!Association to this one!Update your weights!
No association
Try it again
38
Ground truth trajectory
Supervision
( ),Positive example
2
Inverse Reinforcement Learning
t-2
1
2
3
4
t
time axis
t-1
tracked lostGood job!Keep going!No update of the weights
Try it again
39
Ground truth trajectory
Supervision
Active
Tracked
Inactive
Lost
objectdetection
Markov Decision Process for a Single Target
40
Outline
•Markov Decision Process (MDP) for a Single Target
•Online Multi-Object Tracking with MDPs
• Experiments
•Conclusion
41
Ensemble MDPs for Online Multi-Object Tracking
t-2 t
time axis
MDP1
MDP2
MDP3
t-1
42
Step 1: Process tracked targets
t
time axis
MDP1
MDP2
MDP3
t-2 t-1
43
Step 2: Process lost targets
t
time axis
MDP1
MDP2
MDP3
Hungarian algorithm for lost targets
t-2 t-1
44
Step 3: Initialize new targets
t
time axis
MDP1
MDP2
MDP3
Initialize new targets
t-2 t-1
45
Terminate detection
Tracked Lost Tracked
Tracked Lost Tracked
Tracked Tracked Tracked
MDP1
MDP2
MDP3
Online Multi-Object Tracking with MDPs
46
Outline
•Markov Decision Process (MDP) for a Single Target
•Online Multi-Object Tracking with MDPs
• Experiments
•Conclusion
47
Experiments: Dataset
•Multiple Object Tracking Benchmark [1]• 11 training sequences• 11 test sequences• Object detections from the ACF detector [2]
[1] L. Leal-Taixé, A. Milan, I. Reid, S. Roth, and K. Schindler. MOTChallenge 2015: Towards a Benchmark for Multi-Target Tracking. arXiv:1504.01942 [cs], 2015.[2] P. Dollár, R. Appel, S. Belongie, and P. Perona. Fast feature pyramids for object detection. TPAMI, 36(8):1532–1545, 2014. 48
Experiments: Analysis on Validation Set
• Contribution of different components
49
Experiments: Analysis on Validation Set
• Contribution of different components
Active
Tracked
Inactive
Lost
objectdetection
50
MOTA: multiple object tracking accuracy
Experiments: Analysis on Validation Set
• Contribution of different components
Active
Tracked
Inactive
Lost
objectdetection
51
MOTA: multiple object tracking accuracy
Experiments: Analysis on Validation Set
• Contribution of different components
Active
Tracked
Inactive
Lost
objectdetection
52
MOTA: multiple object tracking accuracy
Experiments: Analysis on Validation Set
• Contribution of different components
53
𝜙1( ),Similarity 𝑤1=+⋯
𝜙𝑛( ),𝑤𝑛
+
+
𝑏MOTA: multiple object tracking accuracy
Experiments: Analysis on Validation Set
• Contribution of different components
54
𝜙1( ),Similarity 𝑤1=+⋯
𝜙𝑛( ),𝑤𝑛
+
+
𝑏MOTA: multiple object tracking accuracy
Experiments: Analysis on Validation Set
• Cross-domain tracking
55
MOTA: multiple object tracking accuracy
TUD-Stadtmitte
ETH-Bahnhof
ADL-Rundle-6
KITTI-13
PETS09-S2L1
Trai
nin
g Se
qu
ence
s
Testing sequences
Experiments: Analysis on Validation Set
• Cross-domain tracking
56
TUD-Stadtmitte
ETH-Bahnhof
ADL-Rundle-6
KITTI-13
PETS09-S2L1
MOTA: multiple object tracking accuracy
Trai
nin
g Se
qu
ence
s
Testing sequences
Experiments: Analysis on Validation Set
• Cross-domain tracking
57
TUD-Stadtmitte
ETH-Bahnhof
ADL-Rundle-6
KITTI-13
PETS09-S2L1
MOTA: multiple object tracking accuracy
Trai
nin
g Se
qu
ence
s
Testing sequences
Experiments: Evaluation on Test SetTracker Tracking Learning MOTA
DP_NMS [1] Batch N/A 14.5
TC_ODAL [2] Online Online 15.1
TBD [3] Batch Offline 15.9
SMOT [4] Batch N/A 18.2
RMOT [5] Online N/A 18.6
CEM [6] Online N/A 19.3
SegTrack [7] Batch Offline 22.5
MotiCon [8] Batch Offline 23.1
MDP (Ours) Online Online 30.3
[1] Pirsiavash et al., CVPR’ 11[2] Bae et al., CVPR’14[3] Geiger et al., TPAMI’14
58
MOTA: multiple object tracking accuracy
[4] Dicle et al., ICCV’13[5] Yoon et al., WACV’15[6] Milan et al., TPAMI’14
[7] Milan et al., CVPR’15[8] Leal-Taixé et al., CVPR’14
Tracking Results
59
60
MDP [Ours] MotiCon [Leal-Taixé et al., CVPR’14]
61
MDP [Ours] MotiCon [Leal-Taixé et al., CVPR’14]
62
MDP [Ours] MotiCon [Leal-Taixé et al., CVPR’14]
Outline
•Markov Decision Process (MDP) for a Single Target
•Online Multi-Object Tracking with MDPs
• Experiments
•Conclusion
63
Active
Tracked
Inactive
Lost
Conclusion
Object Detection
Single Object Tracking
Data AssociationTarget Re-identification
64
Code
65
Active
Tracked
Inactive
Lost
Object Detection
Single Object Tracking
Data AssociationTarget Re-identification
66
Thank you!