Upload
others
View
6
Download
0
Embed Size (px)
Citation preview
Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking
Martin Danelljan, Andreas Robinson, Fahad Shahbaz Khan, Michael Felsberg
Discriminative Correlation Filters (DCF)
Applications
• Object recognition
• Object detection
• Object tracking
– Among state-of-the-art since 2014
– KCF, DSST, HCF, SRDCF, Staple …
2Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking
Discriminative Correlation Filters (DCF)
3Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking
Single-resolution feature map
Limitations:Coarse output
scores
DCF Limitations:1. Single-resolution feature map
• Why a problem?
– Combine convolutional layers of a CNN
• Shallow layers: low invariance – high resolution
• Deep layers: high invariance – low resolution
• How to solve?
– Explicit resampling?
• Artefacts, information loss, redundant data
– Independent DCFs with late fusion?
• Sub-optimal, correlations between layers
4Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking
DCF Limitations:2. Coarse output scores
• Why a problem?
– Accurate localization
• Sub-grid (e.g. HOG grid) or sub-pixel accuracy
• More accurate annotations=> less drift
• How to solve?
– Interpolation?
• Which interpolation strategy?
– Interweaving?
• Costly
5Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking
DCF Limitations:3. Coarse labels
• Why a problem?
– Accurate learning
• Sub-grid or sub-pixel supervision
• How to solve?
– Interweaving?
• Costly
– Explicit interpolation of features?
• Artefacts
6Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking
Our Approach: Overview
7Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking
Continuous filters
Continuous output
Multi-resolution features
Multiresolution Features
8Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking
Interpolation Operator
9Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking
Interpolation Operator
10Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking
Convolution Operator
11Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking
Training Loss
12Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking
[Danelljan et al., ICCV 2015]
Training Loss – Fourier Domain
13Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking
Training Loss – Fourier Domain
14Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking
Localization
15Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking
Localization
16Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking
How to set and ?
• Use periodic summation of functions :
• Gaussian function for
• Cubic spline kernel for
• Fourier coefficients with Poisson’s summation formula:
17Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking
Object Tracking Framework: Features
• VGG network
– Pre-trained on ImageNet
– No fine-tuning on application specific data
18Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking
C-COT: Conjugate Gradient
Only need to evaluate
No sparse matrix handling
Finite memory
Warm starting: non-trivial
Tuning of pre-conditioners
SRDCF: Gauss-Seidel
Explicit computation of
Sparse matrix handling
“Infinite” memory
Warm starting: trivial
Object Tracking Framework: Optimization
Solving
19Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking
Object Tracking Framework: Pipeline
• Simple: … – Track – Train – Track – Train – …
• No thresholds
• No hidden “tricks”
20Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking
Experiments: Object Tracking
• 3 datasets: OTB-100, TempleColor, VOT2015
• Layer fusion on OTB:
• Compared to explicit resampling in DCF
– Performance gain: AUC
– Efficiency gain: data size
21Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking
Experiments: OTB (100 videos)
22Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking
Experiments: Temple-Color (128 videos)
23Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking
Experiments: VOT2016
24Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking
[Matej et al., ECCV VOT workshop 2016]
Object Tracking: Speed
• With CNN features: slow ~1 FPS (no GPU)
• With HOG features: ~ real time at SRDCF performance
25Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking
Feature Point Tracking Framework
• Grayscale pixel features,
• Uniform regularization,
26Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking
Experiments: Feature Point Tracking
• The Sintel dataset
27Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking
Feature Point Tracking: KITTI
28Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking
C-COT
KLT
Future Work
• Features
– Fine tuning
– Unsupervised learning
• Optimization
– Warm start in CG (theory and heuristics)
– Preconditioners
– Implementation aspects
– Alternative strategies or update rules
• Further explore of the continuous formulation
29Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking
Conclusions
• Continuous domain learning formulation
– Multi-resolution deep feature maps
– Sub-pixel accurate localization
– Sub-pixel supervision
• Superior results for two applications
– Object tracking
– Feature point tracking
30Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking
Oral and poster: O-4B-03
Friday afternoon(last session)
31Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking
www.liu.se
Martin Danelljan