View
216
Download
0
Category
Tags:
Preview:
Citation preview
CVPR 2013 Diversity Tutorial
Diverse M-Best Solutions inMarkov Random Fields
Dhruv Batra
Virginia Tech
Joint work with: Students: Payman Yadollahpour (TTIC), Abner Guzman-Rivera (UIUC)
Colleagues: Chris Dyer (CMU), Greg Shakhnarovich (TTIC), Pushmeet Kohli (MSRC), Kevin Gimpel (TTIC)
CVPR 2013 Diversity Tutorial
(C) Dhruv Batra 2
CVPR 2013 Diversity Tutorial
Ambiguity Ambiguity Ambiguity
(C) Dhruv Batra 3
?
?
One instance / Two instances?
CVPR 2013 Diversity Tutorial
Problems with MAP
(C) Dhruv Batra 4
Model-Class is Wrong!-- Approximation Error
Not Enough Training Data!-- Estimation ErrorMAP is NP-Hard
-- Optimization ErrorInherent Ambiguity
-- Bayes ErrorMake Multiple Predictions!
Single Prediction = Uncertainty Mismanagement
CVPR 2013 Diversity Tutorial
Multiple Predictions
(C) Dhruv Batra 5
Porway & Zhu, 2011TU & Zhu, 2002Rich History
Sampling
xxx xx x xxx x xx x
CVPR 2013 Diversity Tutorial
Multiple Predictions
(C) Dhruv Batra 6
Flerova et al., 2011Fromer et al., 2009Yanover et al., 2003
M-Best MAP Ideally:
M-Best Modes ✓Porway & Zhu,
2011TU & Zhu, 2002Rich History
Sampling
CVPR 2013 Diversity Tutorial
Multiple Predictions
(C) Dhruv Batra 7
Flerova et al., 2011Fromer et al., 2009Yanover et al., 2003
M-Best MAP Ideally:
M-Best Modes ✓Porway & Zhu,
2011TU & Zhu, 2002Rich History
SamplingOur work: Diverse M-Best in MRFs [ECCV ‘12]
- Don’t hope for diversity. Explicitly encode it.
- Not guaranteed to be modes.
CVPR 2013 Diversity Tutorial
(C) Dhruv Batra 8
Example Result
CVPR 2013 Diversity Tutorial
(C) Dhruv Batra 9
Example Result
Discriminative Re-ranking of Diverse Segmentation
[Yadollahpour et al., CVPR13, Wednesday Poster]
CVPR 2013 Diversity Tutorial
MAP Integer Program
(C) Dhruv Batra 10
kx1
CVPR 2013 Diversity Tutorial
MAP Integer Program
(C) Dhruv Batra 11
kx1
1
0
0
0
CVPR 2013 Diversity Tutorial
MAP Integer Program
(C) Dhruv Batra 12
kx1
0
1
0
0
CVPR 2013 Diversity Tutorial
MAP Integer Program
(C) Dhruv Batra 13
kx1
0
0
1
0
CVPR 2013 Diversity Tutorial
MAP Integer Program
(C) Dhruv Batra 14
kx1
0
0
0
1
CVPR 2013 Diversity Tutorial
MAP Integer Program
(C) Dhruv Batra 15
kx1
0
0
0
1
k2x1
CVPR 2013 Diversity Tutorial
MAP Integer Program
(C) Dhruv Batra 16
kx1
0
0
0
1
k2x1
CVPR 2013 Diversity Tutorial
MAP Integer Program
(C) Dhruv Batra 17
Graphcuts, BP, Expansion, etc
CVPR 2013 Diversity Tutorial
Diverse 2nd-Best
(C) Dhruv Batra 18
MAPDiversity
CVPR 2013 Diversity Tutorial
Diverse M-Best
(C) Dhruv Batra 19
CVPR 2013 Diversity Tutorial
Diverse 2nd-Best
(C) Dhruv Batra 20
Q1: How do we solve DivMBest?
Q2: What kind of diversity functions are allowed?
Q3: How much diversity?
CVPR 2013 Diversity Tutorial
Diverse 2nd-Best
(C) Dhruv Batra 21
Dualize
Diversity-Augmented Score
Primal
CVPR 2013 Diversity Tutorial
Diverse 2nd-Best• Lagrangian Relaxation
(C) Dhruv Batra 22
Diversity-Augmented Score
Dual
Concave (Non-smooth)
Upper-Bound on Div2Best Score
Subgradient Descent
Div2Best score
CVPR 2013 Diversity Tutorial
Diverse 2nd-Best• Lagrangian Relaxation
(C) Dhruv Batra 23
Dualize
Diversity-Augmented EnergyMany ways to solve:
1. Subgradient Ascent. Optimal. Slow.
2. Binary Search. Optimal for M=2. Faster.
3. Grid-search on lambda. Sub-optimal. Fastest.
CVPR 2013 Diversity Tutorial
Theorem Statement• Theorem [Batra et al ’12]: Lagrangian Dual
corresponds to solving the Relaxed Primal:• Based on result from [Geoffrion ‘74]
(C) Dhruv Batra 24
Dual
Relaxed Primal
CVPR 2013 Diversity Tutorial
Effect of Lagrangian Relaxation
(C) Dhruv Batra 25
CVPR 2013 Diversity Tutorial
Effect of Lagrangian Relaxation
(C) Dhruv Batra 26
CVPR 2013 Diversity Tutorial
Effect of Lagrangian Relaxation• [Mezuman et al. UAI13]
(C) Dhruv Batra 27
Pairwise Potential Strength Pairwise Potential Strength
CVPR 2013 Diversity Tutorial
Diverse 2nd-Best
(C) Dhruv Batra 28
Q1: How do we solve DivMBest?
Q2: What kind of diversity functions are allowed?
Q3: How much diversity?
CVPR 2013 Diversity Tutorial
Diversity• [Special Case] 0-1 Diversity M-Best MAP
– [Yanover NIPS03; Fromer NIPS09; Flerova Soft11]
• [Special Case] Max Diversity [Park & Ramanan ICCV11]
• Hamming Diversity
• Cardinality Diversity
• Any Diversity
(C) Dhruv Batra 29
CVPR 2013 Diversity Tutorial
Hamming Diversity
(C) Dhruv Batra 30
0
1
0
0
0 1 0 0
0
1
0
0
1 0 0 0
CVPR 2013 Diversity Tutorial
Hamming Diversity
• Diversity Augmented Inference:
(C) Dhruv Batra 31
CVPR 2013 Diversity Tutorial
Hamming Diversity
• Diversity Augmented Inference:
(C) Dhruv Batra 32
Unchanged. Can still use graph-cuts!
Simply edit node-terms. Reuse MAP machinery!
CVPR 2013 Diversity Tutorial
Diverse 2nd-Best
(C) Dhruv Batra 33
Q1: How do we solve DivMBest?
Q2: What kind of diversity functions are allowed?
Q3: How much diversity?
CVPR 2013 Diversity Tutorial
How Much Diversity?
• Empirical Solution: Cross-Val for
• More Efficient: Cross-Val for
(C) Dhruv Batra 34
CVPR 2013 Diversity Tutorial
Experiments• 3 Applications
– Interactive Segmentation: Hamming, Cardinality (in paper)– Pose Estimation: Hamming– Semantic Segmentation: Hamming
• Baselines:– M-Best MAP (No Diversity)– Confidence-Based Perturbation (No Optimization)
(C) Dhruv Batra 35
CVPR 2013 Diversity Tutorial
Interactive Segmentation• Setup
– Model: Color/Texture + Potts Grid CRF– Inference: Graph-cuts– Dataset: 50 train/val/test images
(C) Dhruv Batra 36
Image + Scribbles Diverse 2nd Best2nd Best MAPMAP
1-2 Nodes Flipped 100-500 Nodes Flipped
CVPR 2013 Diversity Tutorial
Pose Tracking• Setup
– Model: Mixture of Parts from [Park & Ramanan, ICCV ‘11]– Inference: Dynamic Programming– Dataset: 4 videos, 585 frames
(C) Dhruv Batra 37Image Credit: [Yang & Ramanan, ICCV ‘11]
CVPR 2013 Diversity Tutorial
(C) Dhruv Batra 38
CVPR 2013 Diversity Tutorial
Pose Tracking• Chain CRF with M states at each time
(C) Dhruv Batra 39
M BestSolutions
Image Credit: [Yang & Ramanan, ICCV ‘11]
CVPR 2013 Diversity Tutorial
Pose Tracking
(C) Dhruv Batra 40
DivMBest + ViterbiMAP
CVPR 2013 Diversity Tutorial
Pose Tracking
(C) Dhruv Batra 41
1 51 101 151 201 251 30145%
50%
55%
60%
65%
70%
75%
80%
85%
DivMBest (Re-ranked)
[Park & Ramanan, ICCV ‘11] (Re-ranked)
Confidence-based Perturbation (Re-ranked)
13% Gain
Same FeaturesSame Model
#Solutions / Frame
PC
P A
ccur
acy
Better
CVPR 2013 Diversity Tutorial
Machine Translation
Input:Die Regierung will die Folter von “Hexen” unterbinden und gab eine Broschüre heraus
MAP Translation:The government wants the torture of ‘witch’ and gave out a booklet
(C) Dhruv Batra 42
CVPR 2013 Diversity Tutorial
Machine Translation
Input:Die Regierung will die Folter von “Hexen” unterbinden und gab eine Broschüre heraus
5-Best Translations:The government wants the torture of ‘witch’ and gave out a booklet The government wants the torture of “witch” and gave out a booklet The government wants the torture of ‘witch’ and gave out a brochure The government wants the torture of ‘witch’ and gave out a leaflet The government wants the torture of “witch” and gave out a brochure
(C) Dhruv Batra 43
CVPR 2013 Diversity Tutorial
Machine Translation
Input:Die Regierung will die Folter von “Hexen” unterbinden und gab eine Broschüre heraus
Diverse 5-Best Translations:The government wants the torture of ‘witch’ and gave out a bookletThe government wants to stop torture of “witch” and issued a leaflet issuedThe government wants to “stop the torture of” witches and gave out a brochureThe government intends to the torture of “witchcraft” and were issued a leafletThe government is the torture of “witches” stamp out and gave a brochure
(C) Dhruv Batra 44
CVPR 2013 Diversity Tutorial
Machine Translation
Input:Die Regierung will die Folter von “Hexen” unterbinden und gab eine Broschüre heraus
Diverse 5-Best Translations:The government wants the torture of ‘witch’ and gave out a bookletThe government wants to stop torture of “witch” and issued a leaflet issuedThe government wants to “stop the torture of” witches and gave out a brochureThe government intends to the torture of “witchcraft” and were issued a leafletThe government is the torture of “witches” stamp out and gave a brochure
Correct Translation:The government wants to limit the torture of “witches,” a brochure was released
(C) Dhruv Batra 45
Recommended