Finite Systems of Linear Inequalities

Combinatorial Optimization 2

Rumen Andonov

Irisa/Symbiose and University of Rennes 1

9 novembre 2009

Rumen Andonov (Irisa) Combinatorial Optimization 2 9 novembre 2009 1 / 33

Outline

Finite Systems of Linear Inequalities, variants of Farkas’ Lemma

Duality theory in Linear Programming

Application 1 : The shortest path problem (SPP) and its dual

The primal simplex algorithm briefly

The dual simplex algorithm briefly

Application 2 : Max-Flow Min-Cut (MFMC)

Total Unimodularity


Finite Systems of Linear Inequalities

We assume : A ∈ Rm×n,b ∈ Rm×1,c ∈ R1×n,x ∈ Rn×1,y ∈ R1×m

A polytope is conv(X) for some finite set X ⊂ Rn. A polytope can also bedescribed as the solution set of a finite system of linear inequalities, i.e. :

Theorem (Weyl’s theorem :)

If P is a polytope, then

P = {X ∈ Rn :n

∑j=1

aijxj ≤ bi , for i = 1, . . . ,m}

for some m ∈ Z+ and aij ,bi ∈ R.


Finite Systems of Linear Inequalities

We assume : A ∈ Rm×n,b ∈ Rm×1,c ∈ R1×n,x ∈ Rn×1,y ∈ R1×m

A polytope is conv(X) for some finite set X ⊂ Rn. A polytope can also bedescribed as the solution set of a finite system of linear inequalities, i.e. :

Theorem (Weyl’s theorem :)

If P is a polytope, then

P = {X ∈ Rn :n

∑j=1

aijxj ≤ bi , for i = 1, . . . ,m}

for some m ∈ Z+ and aij ,bi ∈ R.

Theorem (of the Alternative for Linear Inequalities :)

{x ∈ Rn : Ax ≤ b} 6= /0 iff {y ∈ Rm+ : yA = 0, yb < 0} = /0


Finite Systems of Linear Inequalities (Farkas’ Lemma )An equivalent result is the following : given A and b,

Theorem (Farkas’ Lemma :)

b) Either {x ∈ Rn+ : Ax = b} 6= /0 or (excl.) {∃y ∈ Rm : yA ≥ 0, yb < 0} 6= /0


Finite Systems of Linear Inequalities (Farkas’ Lemma )An equivalent result is the following : given A and b,

Theorem (Farkas’ Lemma :)


Either b ∈ cone{A1, ...,An} (where Ai are the columns of A), or ∃ a vector ythat makes an acute angle with Ai and an obtuse angle with b, but not both.

��

��

��

��

bb

y

1

A

AA

1

A22


Variants of Farkas’ Lemma

TheoremVariants of Farkas’ Lemma :


a) Either {x ∈ Rn+ : Ax ≤ b} 6= /0 or (excl.) {∃y ∈ Rm

+ : yA ≥ 0, yb < 0} 6= /0

c) Either {x ∈ Rn : Ax ≤ b} 6= /0 or (excl.) {∃y ∈ Rm+ : yA = 0, yb < 0} 6= /0

d) If P = {r ∈ Rn+ : Ar = 0}, either P\{0} 6= /0, or {y ∈ Rm : yA > 0} 6= /0


Duality theory in Linear Programming

Let A ∈ Rm×n,b ∈ Rm×1,c ∈ R1×n, ai is the i-th row and Aj is the j-th col. of A.To any primal (P) is associated its dual (D) by associating a dual variable toeach primal constraint and following the rules :

Primal Dualmax min

variables x constraintsconstraints variables y

objective coefficients c constraint right hand sides cconstraint right hand sides b objective coefficients b

aix ≤ bi yi ≥ 0aix ≥ bi yi ≤ 0aix = bi yi unconstrainedxj ≥ 0 yAj ≥ cj

xj ≤ 0 yAj ≤ cj

xj unconstrained yAj = cj


Duality theory : Canonical Forms

Let be given a primal (P) in a canonical form (i.e.)

(P) z = max{cx |Ax ≤ b,x ∈ Rn,x ≥ 0} (1)

Its dual (D) is as follows.

(D) w = min{yb|yA ≥ c,y ∈ Rm,y ≥ 0} (2)

In general, the following relationships hold :

primal/dual constraint dual/primal constraint (3)

consistent with canonical form ⇐⇒ variable ≥ 0 (4)

reversed from canonical form ⇐⇒ variable ≤ 0 (5)

eqaulity constraint ⇐⇒ variable unrestricted (6)


Duality theory : Primal/Dual relationship (Illustration)

A primal problem and its dual

maxz = 6x1 + x2 + x3

4x1 +3x2 −2x3 = 16x1 −2x2 +9x3 ≥ 92x1 +3x2 +8x3 ≤ 5x1 ≥ 0,x2 ≤ 0,x3 unrestricted

minw = y1 +9y2 +5y3

4y1 +6y2 +2y3 ≥ 63y1 −2y2 +3y3 ≤ 1−2y1 +9y2 +8y3 = 1y1 unrestricted ,y2 ≤ 0,y3 ≥ 0

Lemma :It is easy to verify that the dual of the dual is the primal.


Duality theory in Linear Programming (cont.)Consider the following couple primal-dual (it just just a matter of convenience.(D) is given in the so called standard form to which any LP form can betransformed).

(P) z = max{cx |Ax ≤ b,x ∈ Rn} (7)

(D) w = min{yb|yA = c,y ≥ 0,y ∈ Rm} (8)

We suppose rank(A) = m ≤ n (all redundant equations were removed).

TheoremWeak Duality : If x is feasible to P and y is feasible to D, then cx ≤ yb. Hence

1 if cx = yb, then x and y are optimal

2 if either is unbounded, then the other is infeasible.

Proof. Trivial (cx = yAx ≤ yb).


Duality theory in Linear Programming (cont.)

TheoremStrong Duality : If P and D have feasible solutions, then they have optimalsolutions x,y with cx = yb. If either is infeasible, then the other is eitherinfeasible or unbounded.

Proof. Can be proved using one of the variant of Farkas’ lemma. Here we willdiscuss its geometrical insight only.

Remark :The Theorem of the Alternative for Linear Inequalities can be proven from theStrong Duality Theorem.

Hint : Consider the program

(D0) min{yb : yA = 0,y ∈ Rm+} (9)

First argue that either the optimal objective value for D0 is zero or D0 isunbounded.


Duality theory in Linear Programming (cont.)

Geometrically : Suppose a finite maximum δ (i.e.) δ = cx∗ = max{cx |Ax ≤ b} .

c

��

��

Pa1x*=

a2x*=β2

β1

x*

cx*= δa1

a2

Set A = (a1,a2)t , b = (β1,β2) and let a1x∗ = β1,a2x∗ = β2.

Then ∃ λ1,λ2 ≥ 0 s.t. c = λ1a1 +λ2a2. Hence δ = λ1β1 +λ2β2. This implies :

max{cx |Ax ≤ b} = δ = λ1β1 +λ2β2 ≥ min{yb|yA = c,y ≥ 0} (10)

(since the λi give a feasible solution for the minimum). On the other hand, from theweak duality we have max{cx |Ax ≤ b} ≤ min{yb|yA = c,y ≥ 0}.

Hence max{cx |Ax ≤ b} = min{yb|yA = c,y ≥ 0}.


Visualizing Duality

A physical model of a weighted undirected graph. Each edge is a string of length equalto the edge’s weight, while each node is a knot at which the strings are tied.

Question (How to find the shortest path from S to T ?)


Visualizing Duality



Answer (Just pull S away from T until the gadget is taut.)


Visualizing Duality




Why does it work ?


Visualizing Duality




Why does it work ?Because by pulling S away from T we solve the dual of the shortest-path problem !


Visualizing Duality




Why does it work ?Because by pulling S away from T we solve the dual of the shortest-path problem !

Exercice (Prove the above formally.)


The shortest path problem (SPP) and its dual


Application 1 : The shortest path problem and its dual


Finding the shortest path from S to T is equivalent to pulling S away from T as much aspossible.

We will prove it by writing the primal and the dual problem of the above modeling.


The shortest path problem (SPP)

Given a directed graph G = (V ,U) and weights wj ≥ 0 associated with eacharc uj , find a path from s to t with the minimum total weight.

[3]

a

s

b

t

u1

u2

u3

u4

u5[1]

[2]

[2]

[1]

A =

s +1 +1 0 0 0t 0 0 0 −1 −1a −1 0 +1 +1 0b 0 −1 −1 0 +1

A weighted directed graph and the corresponding node-arc incidence matrix A.The weights are in squared brackets.


SPP : Primal-Dual LP formulation

A path from s to t can be thought of as a flow φ of unit 1 leaving s and enteringt . The primal (P) of (SPP) and its dual (D) where a variable πi is assigned toeach node i , are given below.

(P)

min wφ (11)

Aφ =+1 row s−1 row t

0 otherwise

φ ≥ 0 (12)

(D)

max πs −πt (13)

πi −πj ≤ wij ∀(ij) ∈ U (14)

πi unrestricted ∀i ∈ V (15)


Complementary-Slackness in Duality theory

Let us denote by ai the i-the raw of the matrix A and by Aj its j-the column.Points x ∈ Rn and y ∈ Rm are complementary, with respect to P and D, if

yi(bi −aix) = 0, for i = 1, . . .m (16)

TheoremWeak Complementary-Slackness If feasible solutions x and y arecomplementary, then they are optimal solutions.

Proof.

0 =m

∑i=1

yi(bi −aix) =m

∑i=1

yibi −n

∑j=1

m

∑i=1

yiaijxj = yb− cx

If x and y are complementary and feasible, then yb = cx and by the WeakDuality Theorem they are optimal.


Complementary-Slackness in Duality theory : II

Let us denote by ai the i-the raw of the matrix A and by Aj its j-the column.Points x ∈ Rn and y ∈ Rm are complementary, with respect to P and D, if

yi(bi −aix) = 0, for i = 1, . . .m (17)

TheoremStrong Complementary-Slackness If feasible solutions x and y are optimalsolutions to P and D, respectively, then they are complementary.

Note : Similar considerations hold for the other LP duality equation, i.e.

xj(cj − yAj) = 0, for j = 1, . . .n (18)


Complementary-Slackness in Duality theory : III

TheoremVariant : A pair (x, y) feasible in P and D respectively, is optimal iff.

yi(bi −aix) = 0, for i = 1, . . .m xj(cj − yAj) = 0, for j = 1, . . .n (19)

Geometrically : Suppose δ = max{cx∗|Ax∗ ≤ b} . As we saw then ∃ λ1,λ2 ≥ 0s.t. c = λ1a1 +λ2a2, where a1x∗ = β1,a2x∗ = β2. Note that if ax∗ < β forsome inequality in Ax ≤ b, then the corresponding component in y∗ is zero.

c

��

��

Pa1x*=

a2x*=β2

β1

x*

cx*= δa1

a2


The primal simplex algorithm brieflyGiven the following linear program in standard form :

(P) z = max{cx |Ax = b,x ≥ 0,x ∈ Rn} (20)

Definition

A basis B is a m×m non-singular submatrix of A. Then A = (B,N).

Let x = (xB,xN) and BxB +NxN = b. ⇒ xB +B−1NxN = B−1b = b.xB = B−1b and xN = 0 is called a basic solution of P.

If B−1b ≥ 0, then (xB,xN) is called a basic primal feasible solution and B iscalled a primal feasible basis.

TheoremA point x is an extreme point of the set {x |Ax = b,x ≥ 0} iff it is a basic feasiblesolution.

Theorem ( Fundamental theorem of linear programming)If (P) has a finite optimum then it has an optimal basic feasible solution.


The dual simplex algorithm

The following is the dual of problem (P)

(D) w = min{yb|yA ≥ c,y ∈ Rm} (21)

Let c = (cB,cN) and z = cBxB + cNxN = cBb +(cN − cBB−1N)xN .Set y = cBB−1 ∈ Rm. y is complementary to (xB,xN) sinceyA− c = cBB−1(B,N)− (cB,cN) = (0,cBB−1N − cN) and xN = 0.

Definition

If cBB−1N ≥ cN , then B is called a dual feasible basis.

Theorem

If B is primal and dual feasible then x = (xB,xN) = (B−1b,0) is an optimalsolution to P and y = cBB−1 is an optimal solution to D.


Max-Flow Min-Cut (MFMC)



Let N = (s, t,V ,E ,C) be a flow network with n = |V | nodes, m = |E |arcs, and arc-capacity c(i, j). Let the flow in arc i, j be denoted by φi,j .Flow conservation holds for any node except s, t where

∑e∈Γ+(s)

φe = ∑e∈Γ−(t)

φe = φ0 (flow value) (22)

Find a flow φ = [φ1,φ2, . . .φm] s.t. 0 ≤ φe ≤ ce and maximizing φ0.



Let N = (s, t,V ,E ,C) be a flow network with n = |V | nodes, m = |E |arcs, and arc-capacity c(i, j). Let the flow in arc i, j be denoted by φi,j .Flow conservation holds for any node except s, t where

∑e∈Γ+(s)

φe = ∑e∈Γ−(t)

φe = φ0 (flow value) (22)

Find a flow φ = [φ1,φ2, . . .φm] s.t. 0 ≤ φe ≤ ce and maximizing φ0.

Modeling MFMC as a LP problem

The node-arc incidence matrixA = [ai,j ] i = 1,2, . . . , |V |and j = 1,2, . . . , |E | is defined by ai,j =

+1 if arc ej leaves node i−1 if arc ej enters node i0 otherwise


MFMC : LP formulation

Let (as usual) ai denote the i th row of A.

The flow conservation at a node i (except s, t) is expressed by :

aiφ = 0 (23)

The LP formulation of the MFMC is :

let d ∈ Rn be defined by

di =

−1 i = s+1 i = t0 otherwise

maxφ0 (24)

dφ0 +Aφ = 0 (25)

φ ≤ c (26)

φ ≥ 0 (27)

φ0 ≥ 0 (28)


MFMC : Primal-Dual LP formulation

Assign variables λi , i ∈ V to (30) and variables γi,j ,(i, j) ∈ E to (31). Then weobtain :

(P)

maxφ0 (29)

dφ0 +Aφ = 0 (30)

φ ≤ c (31)

φ ≥ 0 (32)

φ0 ≥ 0 (33)

(D)

min ∑(i,j)∈E

γi,jci,j (34)

λi −λj + γi,j ≥ 0 ∀(i, j) ∈ E (35)

−λs +λt ≥ 1 (36)

γi,j ≥ 0 ∀(i, j) ∈ E (37)


MFMC : Max-Flow Min-Cut Relationships

Definition

An s− t cut is a partition (A, A) of V , s.t. s ∈ A and t ∈ A.The capacity of an s− t cut is C(A, A) = ∑

e∈Γ+(A)

ce.

Theorem

Every s− t cut determines a feasible solution with cost C(A, A) to (D) asfollows :

γi,j =

{

1 (i, j) s.t. i ∈ A, j ∈ A0 otherwise

λi =

{

0 i ∈ A,

1 i ∈ A

Proof : Just by checking. Note that (35) is strict inequality iff i ∈ A and j ∈ A.Note also that −λs +λt = 1.


Max-Flow Min-Cut Theorem

Theorem

The value φ0 is no greater than the capacity C(A, A) of any s− t cut.Furthermore, the value of the maximum flow equals the capacity of theminimum cut, and a flow φ and cut C(A, A) are jointly optimal iff

φi,j = 0 for (i, j) ∈ E such that i ∈ A and j ∈ A (38)

φi,j = ci,j for (i, j) ∈ E such that i ∈ A and j ∈ A (39)

Proof. Trivial, use the complementary slackness.


Total Unimodularity


Total Unimodularity

When solving an LP problem always gives integer solution ?

DefinitionAn integer matrix A is called totally unimodular (TUM) if the determinant ofeach square submatrix of A equals 0,+1, or -1.

P(A) = {x |Ax ≤ b,x ≥ 0,} (40)

Theorem

If A is TUM, then all the vertices of P(A) are integer for any integer vector b.

Proof. First prove the case {x |Ax = b,x ≥ 0,}. Any basic solution

x = B−1b = adj(B)bdet(B) . Then show that (A|I) is TUM.


Total Unimodularity (suite I)

TheoremAn integer matrix A with aij = 0,+1,−1 is TUM if no more than two nonzeroentries appear in any column, and if the rows of A can be partitioned into twosets I1 and I2 s.t. :

1 If a column has two entries of the same sign, their rows are in differentsets ;

2 If a column has two entries of the different signs, their rows are in thesame sets ;

Proof. By induction of the size of submatrices. Let C be a submatrix of size k .For k = 1 it is true. Let k > 1. It is obvious when C has a column of all zeros orwith one nonzero. Let C has two nonzero entries in every column. Then theconditions of the theorem imply :

∑i∈I1

aij = ∑i∈I2

aij for every column j

. i.e. a linear combination of rows is zero, hence det(C) = 0.Rumen Andonov (Irisa) Combinatorial Optimization 2 9 novembre 2009 30 / 33

Total Unimodularity (suite II)

Corollary

Any LP in standard or canonical form whose constraint matrix A is either

1 The node-arc incidence matrix of a directed graph, or

2 The node-edge incidence matrix of an undirected bipartite graph,

has only integer optimal vertices. This includes the LP formulation of shortestpath, max-flow, the transport problem, and weighted bipartite matching.

Proof. The matrices in Case 1 satisfy the condition of the theorem with I2 = /0 ;those in Case 2 with I1 = V and I2 = U, where the bipartite graph isG = (V ,U,E).


Examen June 2009

Exercice 1 : Optimizing an anti-fire systemThe port company COSMAR wishes to improve its anti-fire security by buyingnew anti-fire systems SLIC. The company identifies 7 zones of the port to bespecially protected. These zones are denoted by X on the below figure.

D GFA B C E

NORTH

WEST

1

2

3

5

6

4

X

XX

XXX

X

FIG.: The 7 zones to be protected by the company COSMAR


Optimizing an anti-fire system

COSMAR intends to allocate the SLIC systems on the North and West sides of theport. A SLIC system protects the zones situated on the same row or column. Forexample, a SLIC installed on D covers zones (1,D) et (6,D).It is required that these anti-fire systems cannot be placed on consecutive columns,that at least one SLIC is allocated on row 1 or on column A, and that at least 60% ofthe SLIC are placed on the North side.Questions :

1 Formulate the corresponding linear program that allows to minimize the numberof the anti-fire systems needed to cover the 7 special zones.

2 Since the cost of one SLIC is 8070 euros, COSMAR wishes to buy at most 3 ofthem. For the non-protected zones COSMAR plans to take out insurance againstfire. This insurance costs 2000 euros for each zone on lines 1, 2, 3 and 3000euros for the other lines. Modify the previous linear model in order to minimizethe total cost.

3 Are you aware of software packages permitting to solve such problems ?


Documents

Finite Systems of Linear Inequalities