LP Dualityeaton.math.rpi.edu/faculty/Mitchell/courses/matp6620/... · 2019. 1. 22. · then (D) is also feasible. Further, there exist optimal solutions x⇤ and y⇤ with cTx⇤

LP Duality

John E. Mitchell

Department of Mathematical Sciences

RPI, Troy, NY 12180 USA

January 2016

Mitchell LP Duality 1 / 26

Outline

1 Weak duality

2 Strong duality

3 Complementary slackness

4 Farkas Lemma

5 Visualizing the Farkas Lemma


Outline

LP dual pair

We work with the standard form linear program and its dual:

minx2IRn cT x maxy2IRm bT ysubject to Ax = b (P) subject to AT y c (D)

x � 0

where c 2 IRn, b 2 IRm, and A 2 IRm⇥n. The results extend to any

primal-dual pair, including

minx2IRn cT x maxy2IRm bT ysubject to Ax � b (P) subject to AT y c (D)

x � 0 y � 0


Weak duality

Outline

1 Weak duality

2 Strong duality


4 Farkas Lemma



Weak duality

Weak duality

Theorem

Weak duality: If x is feasible in (P) and y is feasible in (D) thencT x � bT y.

Proof.

We have

bT y = (Ax)T y = xT AT y xT c = cT x

where the inequality follows because x � 0 and AT y c.


Weak duality

Weak duality

Theorem

Weak duality: If x is feasible in (P) and y is feasible in (D) thencT x � bT y.

Proof.

We have

bT y = (Ax)T y = xT AT y xT c = cT x

where the inequality follows because x � 0 and AT y c.


Arab, 5 7 0 AtfE c

Weak duality

Sufficient optimality criterion

As an immediate consequence of the weak duality theorem, we have

the following sufficient condition for optimality:

Theorem

Sufficient optimality criterion: If x is primal feasible and y is dualfeasible and cT x = bT y then x is optimal in (P) and y is optimal in (D).


Strong duality

Outline

1 Weak duality

2 Strong duality


4 Farkas Lemma



Strong duality

Strong duality

This condition is also necessary:

Theorem

Strong duality theorem: If (P) is feasible with a finite optimal valuethen (D) is also feasible. Further, there exist optimal solutions x⇤ andy⇤ with cT x⇤ = bT y⇤.

We will prove this theorem later. In particular, we will use the simplex

algorithm to show how a dual optimal solution can be constructed with

cT x⇤ = bT y⇤.


Strong duality

Possible states of an optimization problem

A general optimization problem can be in one of four possible states:

Infeasible

Feasible with an unbounded optimal value.

Feasible with a finite optimal value that is attained.

Feasible with a finite optimal value that is not attained.

It follows from the strong duality theorem that this last case is not

possible for a linear program. The strong duality theorem also implies

that the state of a linear program tells us the state of its dual linear

program. So a primal-dual pair can only be in one of 4 possible states,

rather than 4 ⇥ 4 = 16 possible states.


Strong duality



Infeasible










Strong duality



Infeasible










Strong duality



Infeasible










r u i n e - ×i t . x 2 0

I

Strong duality

Possible states of an LP pair

In particular, we have the following consequence of the strong duality

theorem:

Theorem

The following are mutually exclusive and exhaustive possibilities for(P) and (D):

(P) and (D) are both infeasible.(D) is infeasible and (P) has an unbounded optimal value.(P) is infeasible and (D) has an unbounded optimal value.Both (P) and (D) are feasible, and they have the same optimalvalue.


Strong duality



theorem:

Theorem




Strong duality



theorem:

Theorem




Strong duality



theorem:

Theorem




Strong duality



theorem:

Theorem




Complementary slackness

Outline

1 Weak duality

2 Strong duality


4 Farkas Lemma





Theorem

A pair of primal and dual feasible solutions are optimal to theirrespective problems in a primal-dual pair of LPs if and only if

whenever these variables make a slackvariable in one problem strictly positivethe value of the associated nonnegativevariable in the other is zero.

9>>=

>>;

complementaryslackness


m m × ,-12×12+34, (p)Optimal:

S t . × , -1×2++3=1 ×*=(1,0,O)x i2 0 Value : I

5 1=0m a x ys e . y@tight.

Slacker

y e a(D) Optimal:

yes}slack>o y-4=1,f - O , value: 1 .

y free s , >0

Complementary,lactic,,: s , >o , s , >oS i x t o

⇒ x x x ,-Oft i


Proof of complementary slackness

We prove this for the standard pair (P) and (D). We can define the

vector of dual slacks s = c � AT y 2 IRn for any y 2 IRm. Note that the

duality gap is

cT x � bT y = cT x � (Ax)T y = cT x � xT AT y = cT x � (AT y)T x

= (c � AT y)T x = sT x =nX

i=1

sixi

for any y 2 IRm and x 2 IRn with Ax = b.



Proof of complementary slackness continued

Note that if x and y are feasible in their respective problems then x � 0

and s � 0, so sT x � 0.

If the points are optimal then cT x � bT y = 0 soPn

i=1sixi = 0, so each

component sixi = 0, since they must all be nonnegative. So either

si = 0 or xi = 0 for each component i . This is complementary

slackness.

If complementary slackness holds then either xi = 0 or si = 0 for each

component i , so sT x = 0 so cT x = bT y so the points are optimal. ⇤



Complementary slackness for standard primal-dual

pair

Exercise:

Consider the primal-dual pair

minx cT x maxy bT ysubject to Ax � b (P) subject to AT y c (D)

x � 0 y � 0

and let s = c � AT y , w = Ax � b. Show cT x � bT y = xT s + yT w .


T TExisi S yi w i

Farkas Lemma

Outline

1 Weak duality

2 Strong duality


4 Farkas Lemma



Farkas Lemma

Farkas Lemma

Theorem

Farkas Lemma: Let A 2 IRm⇥n and b 2 IRm. Exactly one of thefollowing two systems has a solution:

(I): Ax = b, x � 0. (II): AT y 0, bT y > 0.


Farkas Lemma

Farkas Lemma

Theorem

Farkas Lemma: Let A 2 IRm⇥n and b 2 IRm. Exactly one of thefollowing two systems has a solution:

(I): Ax = b, x � 0. (II): AT y 0, bT y > 0.


Farkas Lemma

Proof of Farkas Lemma

Proof.

Prove using LP duality. Set up the primal-dual pair (P) and (D) with

c = 0. Note that y = 0 is feasible in (D), so (D) either has finite

optimal value or unbounded optimal value. Note also that (P) is either

infeasible or has optimal value equal to zero.

If (I) is consistent then (P) has optimal value 0 so (D) also has optimal

value 0, so there is no y 2 IRm with both AT y 0 and bT y > 0, so (II)is inconsistent.

If (I) is inconsistent then (P) is infeasible so (D) has unbounded

optimal value, so there exists y with AT y 0 and bT y > 0, so (II) is

consistent.


( I ) Axeb.→ O (II) A' je 0 ,big>o(p) m i n O x (D) m a x bey

s e . Ax-b s - t . AlysO÷( I ) consistent ⇒ (P)feasible,optimal valueof O

⇒ (D)feasible, optimalvalue0 ⇒ ( I l ) inconsistent.

-( I ) inconsistent ⇒ (P)

infeasible

⇒ either: (D) infeasible-(D)alwan,feasible:

° ' : ( D ) unboundedt a key-OER"

⇒ (D) unbounded ⇒ I T will Atyeoandbig>

o .⇒ ( I )consistent.

Visualizing the Farkas Lemma

Outline

1 Weak duality

2 Strong duality


4 Farkas Lemma




The columns of A

Let

A =

1 3 2

4 2 1

�.

For what vectors b 2 IR2 is System I consistent?

For what b 2 IR2 is System II consistent?



Consistency of System I

For System I, we require b = Ax for some x � 0.

We can express Ax as follows:

Ax =

1 3 2

4 2 1

�2

4x1

x2

x3

3

5 =

x1 + 3x2 + 2x3

4x1 + 2x2 + x3

�

= x1

1

4

�+ x2

3

2

�+ x3

2

1

�

Thus, System I is consistent if and only if b is a nonnegative linearcombination of the columns of A. This is a cone in IRm.



Nonnegative combinations of the columns of A

b1

b2

0

4

8

4 8

(1,4)

(3,2)

(2,1)

A =

1 3 2

4 2 1

�I: Ax = b, x � 0




b1

b2

0

4

8

4 8

(1,4)

(3,2)

(2,1)

A =

1 3 2

4 2 1

�I: Ax = b, x � 0

x =

2

40.5

0

0

3

5

Ax =

0.5

2

�(0.5,2)




b1

b2

0

4

8

4 8

(1,4)

(3,2)

(2,1)

A =

1 3 2

4 2 1

�I: Ax = b, x � 0

x =

2

4t0

0

3

5

Ax =

t

4t

�

t � 0




b1

b2

0

4

8

4 8

(1,4)

(3,2)

(2,1)

A =

1 3 2

4 2 1

�I: Ax = b, x � 0

x =

2

40

0

2

3

5

Ax =

4

2

�

(4,2)




b1

b2

0

4

8

4 8

(1,4)

(3,2)

(2,1)

A =

1 3 2

4 2 1

�I: Ax = b, x � 0

x =

2

40

0

s

3

5

Ax =

2s

s

�

s � 0




b1

b2

0

4

8

4 8

(1,4)

(3,2)

(2,1)

A =

1 3 2

4 2 1

�I: Ax = b, x � 0

x =

2

40.51.5

2

3

5

Ax =

9

7

�

(9,7)




b1

b2

0

4

8

4 8

System I

consistent

for b here

(1,4)

(3,2)

(2,1)

A =

1 3 2

4 2 1

�I: Ax = b, x � 0



A polar cone

System II considers the y 2 IRm satisfying AT y 0. We have:

{y 2 IRm : AT y 0} =

8<

:y 2 IR2 :

2

41 4

3 2

2 1

3

5

y1

y2

�

2

40

0

0

3

5

9=

;

=

(y 2 IR2 :

1

4

�T y1

y2

� 0,

3

2

�T y1

y2

� 0,

2

1

�T y1

y2

� 0

)

so we consider the set of vectors y that make an angle of at least ⇡/2

with each column of A. This is another cone in IRm.

If b is a nonzero vector in this cone then one solution to System II is to

take y = b, since then bT y = bT b > 0.


f.i t :#I


Picturing the polar cone

y1

y2

0

4

�4

�6

(1,4)

(3,2)

(2,1)

II: AT y 0, bT y > 0

A =

1 3 2

4 2 1

�




y1

y2

0

4

�4

�6

y1 + 4y2 0

(1,4)

(3,2)

(2,1)


A =

1 3 2

4 2 1

�




y1

y2

0

4

�4

�6

2y1 + y2 0

(1,4)

(3,2)

(2,1)


A =

1 3 2

4 2 1

�




y1

y2

0

4

�4

�6

3y1 + 2y2 0

(1,4)

(3,2)

(2,1)


A =

1 3 2

4 2 1

�




y1

y2

0

4

�4

�6

y1 + 4y

2 = 0

2y1 +

y2=

0

{y : AT y 0}

y1 + 4y2 0

3y1 + 2y2 0

2y1 + y2 0

(1,4)

(3,2)

(2,1)


A =

1 3 2

4 2 1

�




y1

y2

0

4

�4

�6

y1 + 4y

2 = 0

2y1 +

y2=

0

{y : AT y 0}

y1 + 4y2 0

3y1 + 2y2 0

2y1 + y2 0

System II

consistent:

take y = b

(1,4)

(3,2)

(2,1)


A =

1 3 2

4 2 1

�


IE


Other choices of b

If b is in neither cone then there is a point y on the boundary of the

second cone that makes positive inner product with b. This y satisfies

the conditions of System II.

We can graph the problem in IRm. The columns of A are points in IRm,

and the two cones are subsets of IRm.



The cones

b1

b2

0

4

�4

4�6

System I

consistent

y1 + 4y

2 = 0

2y1 +

y2=

0

{y : AT y 0}

System II

consistent:

take y = b

(1,4)

(3,2)

(2,1)



The cones

b1

b2

0

4

�4

4�6

System I

consistent

y1 + 4y

2 = 0

2y1 +

y2=

0

{y : AT y 0}

System II

consistent:

take y = b

System II

consistent:

take y = (�4, 1)so bT y > 0

(-4,1)

(1,4)

(3,2)

(2,1)


• b

¥-97.b


The cones

b1

b2

0

4

�4

4�6

System I

consistent

y1 + 4y

2 = 0

2y1 +

y2=

0

{y : AT y 0}

System II

consistent:

take y = b

System II

consistent:

take y = (�4, 1)so bT y > 0

(-4,1)

System II

consistent:

take y = (1,�2)so bT y > 0

(1,-2)

(1,4)

(3,2)

(2,1)


Documents

LP Dualityeaton.math.rpi.edu/faculty/Mitchell/courses/matp6620/... · 2019. 1. 22. · then (D) is also feasible. Further, there exist optimal solutions x⇤ and y⇤ with cTx⇤