Diverse M-Best Solutions in Markov Random Fieldsdbatra/cvpr13diversity... · CVPR 2013 Diversity Tutorial Machine Translation Input:! Die Regierung will die Folter von “Hexen”

CVPR 2013 Diversity Tutorial

Diverse M-Best Solutions in Markov Random Fields

Dhruv Batra Virginia Tech

Joint work with: Students: Payman Yadollahpour (TTIC), Abner Guzman-Rivera (UIUC) Colleagues: Chris Dyer (CMU), Greg Shakhnarovich (TTIC),

Pushmeet Kohli (MSRC), Kevin Gimpel (TTIC)


(C) Dhruv Batra 2


Ambiguity Ambiguity Ambiguity

(C) Dhruv Batra 3

?

?

One instance / Two instances?


Problems with MAP

(C) Dhruv Batra 4

Model-Class is Wrong! -- Approximation Error

Not Enough Training Data! -- Estimation Error MAP is NP-Hard

-- Optimization Error Inherent Ambiguity

-- Bayes Error Make Multiple Predictions!

Single Prediction = Uncertainty Mismanagement


Multiple Predictions

(C) Dhruv Batra 5

XMAP

X

P (X )

Porway & Zhu, 2011!TU & Zhu, 2002!Rich History!

Sampling

x x x x x x x x x x x x x



(C) Dhruv Batra 6

Flerova et al., 2011!Fromer et al., 2009!Yanover et al., 2003!

M-Best MAP Ideally:

M-Best Modes ✓!

XMAP

X

P (X )

Porway & Zhu, 2011!TU & Zhu, 2002!Rich History!

Sampling



(C) Dhruv Batra 7

Flerova et al., 2011!Fromer et al., 2009!Yanover et al., 2003!

M-Best MAP Ideally:

M-Best Modes ✓!Porway & Zhu, 2011!

TU & Zhu, 2002!Rich History!

Sampling

XMAP

X

P (X )

Our work: Diverse M-Best in MRFs [ECCV ‘12]

-  Don’t hope for diversity. Explicitly encode it.

-  Not guaranteed to be modes.


(C) Dhruv Batra 8

CRF

. . .

Diverse Segmentations

Re-ranked List

Top Solution

Re-ranker α�ψ(x,y)

Example Result


(C) Dhruv Batra 9

CRF

. . .

Diverse SegmentationsStage 1

Re-ranker

Re-ranked List

Top Solution

α�ψ(x,y)

. . .

Stage 2

Example Result

Discriminative Re-ranking of Diverse Segmentation

[Yadollahpour et al., CVPR13, Wednesday Poster]


MAP Integer Program

(C) Dhruv Batra 10

θi(s)

kx1

µi(s)�

i

maxy

S(y) = θi(yi) +�

(i,j)

θij(yi, yj)


MAP Integer Program

(C) Dhruv Batra 11

θi(s)

kx1

1

0

0

0

�

i

maxy

S(y) = +�

(i,j)

θij(yi, yj)


MAP Integer Program

(C) Dhruv Batra 12

θi(s)

kx1

0

1

0

0

�

i

maxy

S(y) = +�

(i,j)

θij(yi, yj)


MAP Integer Program

(C) Dhruv Batra 13

θi(s)

kx1

0

0

1

0

�

i

maxy

S(y) = +�

(i,j)

θij(yi, yj)


MAP Integer Program

(C) Dhruv Batra 14

θi(s)

kx1

0

0

0

1

�

i

maxy

S(y) = +�

(i,j)

θij(yi, yj)


MAP Integer Program

(C) Dhruv Batra 15

θi(s)

kx1

0

0

0

1

θij(s, t)�

i

+�

(i,j)

k2x1

µij(s, t)maxy

S(y) =


MAP Integer Program

(C) Dhruv Batra 16

θi(s)

kx1

0

0

0

1

�

i

+�

(i,j)

XMAP

X

P (X )

maxµ∈C

�

i

θi · µi +�

(i,j)

θij · µij

s.t. µi(·),µij(·) ∈ {0, 1}

θij(s, t)

k2x1

µij(s, t)maxy

S(y) =


MAP Integer Program

(C) Dhruv Batra 17

Graphcuts, BP, Expansion, etc

XMAP

X

P (X )

maxµ∈C

�

i

θi · µi +�

(i,j)

θij · µij

s.t. µi(·),µij(·) ∈ {0, 1}


Diverse 2nd-Best

(C) Dhruv Batra 18

∆(µ,µ(1)) ≥ k

XMAP

X

P (X )

MAP

∆(µ,µ(1)) ≥ k

Diversity

maxµ∈C

�

i

θi · µi +�

(i,j)

θij · µij

s.t. µi(·),µij(·) ∈ {0, 1}


Diverse M-Best

(C) Dhruv Batra 19

∆(µ,µ(1)) ≥ k

∆(µ,µ(2)) ≥ k

∆(µ,µ(M−1)) ≥ k

XMAP

X

P (X )

∆(µ,µ(M−1)) ≥ k

∆(µ,µ(1)) ≥ k

∆(µ,µ(2)) ≥ k

maxµ∈C

�

i

θi · µi +�

(i,j)

θij · µij

s.t. µi(·),µij(·) ∈ {0, 1}


Diverse 2nd-Best

(C) Dhruv Batra 20

∆(µ,µ(1)) ≥ k

Q1: How do we solve DivMBest?

Q2: What kind of diversity functions are allowed?

Q3: How much diversity?

maxµ∈C

�

i

θi · µi +�

(i,j)

θij · µij

s.t. µi(·),µij(·) ∈ {0, 1}


Diverse 2nd-Best

(C) Dhruv Batra 21

Dualize

Diversity-Augmented Score

Primal

∆(µ,µ(1)) ≥ k

maxµ∈C

�

i

θi · µi +�

(i,j)

θij · µij

s.t. µi(·),µij(·) ∈ {0, 1}

+λ ·�∆(µ,µ(1))− k

�

XMAP

X

P (X )

S(y) + Div(y,y(1))


Diverse 2nd-Best •  Lagrangian Relaxation

(C) Dhruv Batra 22

Diversity-Augmented Score

Dual

Concave (Non-smooth)

Upper-Bound on Div2Best Score

f(λ) =

Subgradient Descent

∇f(λ(0))

λ(0)

∇f(λ(1))

λ(1)

maxµ∈C

�

i

θi · µi +�

(i,j)

θij · µij

s.t. µi(·),µij(·) ∈ {0, 1}

+λ ·�∆(µ,µ(1))− k

�

minλ≥0

f(λ)

∇f(λ) =�∆(µλ,µ

(1))− k�

λ

Div2Best score

f(λ)


Diverse 2nd-Best •  Lagrangian Relaxation

(C) Dhruv Batra 23

Dualize

Diversity-Augmented Energy

−λ ·�∆(µ,µ(1))− k

�f(λ) = min

µ∈C

�

i

θi · µi +�

(i,j)

θij · µij

s.t. µi(·),µij(·) ∈ {0, 1}∆(µ,µ(1)) ≥ k

f(λ) =

Many ways to solve:

1.  Subgradient Ascent. Optimal. Slow.

2. Binary Search. Optimal for M=2. Faster.

3. Grid-search on lambda. Sub-optimal. Fastest.


Theorem Statement •  Theorem [Batra et al ’12]: Lagrangian Dual

corresponds to solving the Relaxed Primal: •  Based on result from [Geoffrion ‘74]

(C) Dhruv Batra 24

Dual

maxµ

�

i

θi · µi +�

ij

θij · µij

s.t. µ ∈ Co�µi(·),µij(·) ∈ {0, 1} | µ ∈ C

�

∆(µ,µ(1)) ≥ k

Relaxed Primal

minλ≥0

LagrangianDual(λ)


Effect of Lagrangian Relaxation

(C) Dhruv Batra 25

µ(1)

µ(2)

µ(3) µ(4)

µ(5)


Effect of Lagrangian Relaxation

(C) Dhruv Batra 26

µ(1)

µ(2)

µ(3) µ(4)

µ(5)


Effect of Lagrangian Relaxation •  [Mezuman et al. UAI13]

(C) Dhruv Batra 27

Pairwise Potential Strength Pairwise Potential Strength


Diverse 2nd-Best

(C) Dhruv Batra 28

∆(µ,µ(1)) ≥ k




minµ∈C

�

i

θi · µi +�

(i,j)

θij · µij

s.t. µi(·),µij(·) ∈ {0, 1}


Diversity •  [Special Case] 0-1 Diversity M-Best MAP

–  [Yanover NIPS03; Fromer NIPS09; Flerova Soft11]

•  [Special Case] Max Diversity [Park & Ramanan ICCV11]

•  Hamming Diversity

•  Cardinality Diversity

•  Any Diversity

(C) Dhruv Batra 29

=⇒

=⇒

maxµ∈C

S(µ) + λ∆(µ,µ(1))


Hamming Diversity

(C) Dhruv Batra 30

∆(µ,µ(1)) = −�

i∈Vµi · µ(1)

i

0

1

0

0

0 1 0 0 = 1

0

1

0

0

1 0 0 0 = 0


Hamming Diversity

•  Diversity Augmented Inference:

(C) Dhruv Batra 31

θ̃i

∆(µ,µ(1)) = −�

i∈Vµi · µ(1)

i

maxµ∈C

�

i

θi · µi +�

(i,j)

θij · µij + λ∆(µ,µ(1))

=�

i

�θi − λµ(1)

i

�· µi +

�

(i,j)

θij · µij


Hamming Diversity

•  Diversity Augmented Inference:

(C) Dhruv Batra 32

∆(µ,µ(1)) = −�

i∈Vµi · µ(1)

i

Unchanged. Can still use graph-cuts!

Simply edit node-terms. Reuse MAP machinery!

for i = 1,2,. . .,n

θi[x(1)i ] -= λ

endfor

x(2) = Find MAP(θi, θij)


Diverse 2nd-Best

(C) Dhruv Batra 33

∆(µ,µ(1)) ≥ k




minµ∈C

�

i

θi · µi +�

(i,j)

θij · µij

s.t. µi(·),µij(·) ∈ {0, 1}


How Much Diversity?

•  Empirical Solution: Cross-Val for

•  More Efficient: Cross-Val for

(C) Dhruv Batra 34

X

P (X )

λ

k


Experiments •  3 Applications

–  Interactive Segmentation: Hamming, Cardinality (in paper) –  Pose Estimation: Hamming –  Semantic Segmentation: Hamming

•  Baselines: –  M-Best MAP (No Diversity) –  Confidence-Based Perturbation (No Optimization)

(C) Dhruv Batra 35


Interactive Segmentation •  Setup

–  Model: Color/Texture + Potts Grid CRF –  Inference: Graph-cuts –  Dataset: 50 train/val/test images

(C) Dhruv Batra 36

Image + Scribbles Diverse 2nd Best 2nd Best MAP MAP

1-2 Nodes Flipped 100-500 Nodes Flipped


Pose Tracking •  Setup

–  Model: Mixture of Parts from [Park & Ramanan, ICCV ‘11] –  Inference: Dynamic Programming –  Dataset: 4 videos, 585 frames

(C) Dhruv Batra 37 Image Credit: [Yang & Ramanan, ICCV ‘11]


(C) Dhruv Batra 38


Pose Tracking •  Chain CRF with M states at each time

(C) Dhruv Batra 39

M Best Solutions

Image Credit: [Yang & Ramanan, ICCV ‘11]


Pose Tracking

(C) Dhruv Batra 40

DivMBest + Viterbi MAP


Pose Tracking

(C) Dhruv Batra 41

45%

50%

55%

60%

65%

70%

75%

80%

85%

1 51 101 151 201 251 301

DivMBest (Re-ranked)

[Park & Ramanan, ICCV ‘11] (Re-ranked)

Confidence-based Perturbation (Re-ranked)

13% Gain

Same Features Same Model

#Solutions / Frame

PC

P A

ccur

acy

Better


Machine Translation Input: Die Regierung will die Folter von “Hexen” unterbinden und gab eine Broschüre heraus MAP Translation: The government wants the torture of ‘witch’ and gave out a booklet

(C) Dhruv Batra 42


Machine Translation Input: Die Regierung will die Folter von “Hexen” unterbinden und gab eine Broschüre heraus 5-Best Translations: The government wants the torture of ‘witch’ and gave out a booklet The government wants the torture of “witch” and gave out a booklet The government wants the torture of ‘witch’ and gave out a brochure The government wants the torture of ‘witch’ and gave out a leaflet The government wants the torture of “witch” and gave out a brochure

(C) Dhruv Batra 43


Machine Translation Input: Die Regierung will die Folter von “Hexen” unterbinden und gab eine Broschüre heraus Diverse 5-Best Translations: The government wants the torture of ‘witch’ and gave out a booklet The government wants to stop torture of “witch” and issued a leaflet issued The government wants to “stop the torture of” witches and gave out a brochure The government intends to the torture of “witchcraft” and were issued a leaflet The government is the torture of “witches” stamp out and gave a brochure

(C) Dhruv Batra 44


Machine Translation Input: Die Regierung will die Folter von “Hexen” unterbinden und gab eine Broschüre heraus Diverse 5-Best Translations: The government wants the torture of ‘witch’ and gave out a booklet The government wants to stop torture of “witch” and issued a leaflet issued The government wants to “stop the torture of” witches and gave out a brochure The government intends to the torture of “witchcraft” and were issued a leaflet The government is the torture of “witches” stamp out and gave a brochure

Correct Translation: The government wants to limit the torture of “witches,” a brochure was released

(C) Dhruv Batra 45

Documents

Diverse M-Best Solutions in Markov Random Fieldsdbatra/cvpr13diversity... · CVPR 2013 Diversity Tutorial Machine Translation Input:! Die Regierung will die Folter von “Hexen”