Oxford Reading GroupOxford Reading Group

Phil Torr (Oxford Brookes)

Outline of LectureOutline of Lecture

Motivation: What problems may be solved by Semi Definite Programming SDP: segmentation, matching, classification.

What is SDP?

How can it be implemented.

Linear ProgrammingLinear Programming

Includes many problemsGraph cut

Segmentationstereo

Shortest pathTrackingObject recognitionStereo

Some L1 regressions

SDPSDP

Has been described as the most exciting mathematical programming development in the 1990’s (Robert M Freund; MIT).

The simplex method etc from LP can be generalized to SDP.

Classification by EllipsoidsClassification by Ellipsoids

Rosen [95].Constrain A matrices to be PSD.

Another Application: Another Application:

Recall Min Cut ProblemRecall Min Cut Problem

s-t graph cut

A graph with two terminals S and T

“source”

S T

“sink”

• Cut cost is a sum of severed edge weights • Minimum cost s-t cut can be found in polynomial time

Goal: divide the graph into two parts separating red and blue nodes

Segmentation - ModelSegmentation - Model

• Input: Image consisting of pixels

• Output: Segmentation of pixels

• Color model

• Coherence or Edge model

Input Output

5 Hard Degmentation – Probablistic Framework GRABGRABCUTCUT

Color ModelColor Model

Assignment variable:

Which mixture component does a pixel belong to?

7 Hard Degmentation – Probablistic Framework GRABGRABCUTCUT

GraphCut for InferenceGraphCut for Inference

Cut: A collection of edges which separates the Source from the Sink

MinCut: The cut with minimum weight (sum of edge weights)

Solution: Global optimum (MinCut) in polynomial time

Image

Sink

Source

Foreground

Background

Cut

21 Hard Degmentation – Probablistic Framework GRABGRABCUTCUT

22 Hard Degmentation – Probablistic Framework GRABGRABCUTCUT

GraphCut for InfernceGraphCut for Infernce

MinCut minimizes the energy of the MRF:

Image

Sink

Source

Foreground

Background

Cut

constant

MinCutMinCut

Edge weights must all be positive,

Then soluble in polynomial time, by max flow algorithm.

Weights are capacities therefore negative capacity does not make sense.

MaxCutMaxCut

Edge weights may be negative,

Note MaxCut and MinCut are same problem, however term MaxCut used when weights can be negative and positive.

MaxCut NP complete.

Negative weights, MaxCutNegative weights, MaxCut

Edge weights may be negative,

Problem is NP-hard

With SDP, an approximation ratio of 0.878 can be obtained! (Goemans-Williamson ’95), i.e. within 13% of the global energy minimum.

Why negative weightsWhy negative weights

In example above the MinCut produces an odd segmentation, negative weights encode the idea of repulsive force that might yield a better segmentation.

Pop outPop out

Maybe need attractive and repulsive forces to get Gestalt effects:

MaxCut Integer ProgramMaxCut Integer Program

GraphCut

LaplacianLaplacian

∑ wij (1 - xi xj ) = x┬ (diag(W 1) - W) x

Laplacian of graph

MaxCut Integer ProgramMaxCut Integer Program

Laplacian (semi positive definite)

Min/Max Cut minimizes this integer program (cf Hopfield network):

Solving via relaxationSolving via relaxation

Problem above NP complete

Some NP complete problems can be approximated using a “relaxation”, e.g. from binary to continuous variables etc.

Next semi definite programming is explained and then it is shown what relaxation can be used to help solve MaxCut.

Solving via relaxationSolving via relaxation

Keuchel et al 2002; suggest adding a constraint

Where e = (1, … 1) to favour partitions with equal numbers of nodes.

Solving via Solving via relaxationrelaxation

The vector e corresponds to no cut so is an eigenvector of L with eigenvalue 0, a natural relaxation of the problem is to drop the integer constraint and solve for the second smallest eigenvector of L (the Fiedler vector, c.f. Shi & Malik)

Eigenvector may or may not be binary.

Linear Programming (Linear Programming (LPLP))

Canonical form, inequalities removed by introduction of surplus variables.

LP ExampleLP Example

Semidefinite Programming Semidefinite Programming

If X and Ai are diagonal this is a linear program,<X,Y> also used for inner produce of two matrices.

Positive Semidefinite MatricesPositive Semidefinite Matrices

W is Gram matrix, not weightW is Gram matrix, not weight

Semi Definite ProgrammingSemi Definite Programming

Semi Definite ProgrammingSemi Definite Programming

If X diagonal then reduces to LP

The feasible solution space of SDP is convex.

Polynomial time exact solution.

Note most non trivial applications of SDP are equivalent to minimizing the sum of the first few eigenvalues of X with respect to some linear constraints on X.

Recall:Recall:

Min/Max Cut minimizes this integer program:

SDP relaxation of MaxCutSDP relaxation of MaxCut

The relaxation is to relax the rank one constraint for X; allowing X to be real valued. Bad news, many more variables.

Binary variables Binary variables

}1,1{ s.t.2

1Max

i

jiij

x

xxw

Graph Vertices: Unit VectorsGraph Vertices: Unit Vectors

1||||, s.t.2

1Max

in

i

jiij

vRv

vvw

AlgorithmAlgorithm

X is a continuous matrix, recovered by SDP.Recover Gram matrix V, X = VVT by Cholesky decomposition matrix, each vertex represented by a vector vi on unit sphere.Choose random hyperplane to bisect unit sphere and define cut.

RecallRecall

ji vv ji |v| cos (a)|v |

Vertices on same side of cut nearby on unit sphere yields large dot product, vertices far apart yield small (negative) dot product.

An SDP Relaxation of An SDP Relaxation of MAX CUT – MAX CUT –

Geometric intuitionGeometric intuition

Embed the vertices of the graph on the unit sphere such that vertices that are joined by edges are far apart.

Random separationRandom separation

vi

vj

Algorithm AnalysisAlgorithm Analysis

The probability that two vectors are separated by a The probability that two vectors are separated by a random hyperplane:random hyperplane:

Algorithm AnalysisAlgorithm Analysis

Calculate expected value of cut.

Note value of relaxation exceeds cost of MaxCut (as it lives in a less constrained space).

Note the following identity:

11

1

ratio min2

1 = 0.8785Cos ( ) 6..

x x

x

1

1

1

1

exp

sdp

ratio min

1

2

2

1

cos

0.8785

( )

cos (.

)6. .

ii

jj

ijj

x

i

w

v v

x

v

w

v

x

Expected Value of CutExpected Value of Cut

Classification by EllipsoidsClassification by Ellipsoids

Rosen [95].Constrain A matrices to be PSD.

Another Application: Another Application:

Set of matches m, which can be on or off, so that m = 0,1

Change of variable k = 2m -1 changes this to the max cut problem.