LP Introduction

Linear Programming:

Theory and Applications

Linear Programming (LP) Problem

• The problem of optimizing (minimize or

maximize) a linear objective function

subject to linear equality and inequality

constraints

• min f(x)

s.t. ci(x) = 0, i = 1, …, p

ci(x) ≤ 0, i = p+1, …, m

General Form LP

},...,2,1{,

},...,2,1{,

..

min

mppibxa

pibxa

ts

xc

i

T

i

i

T

i

T

• x is the vector of decision variables: – x = [x1 x2 … xn]

T

• ai’s and c are vectors of dimension n

• bi’s are real numbers

Standard Form LP

• A standard form LP is in the following format:

where

0

..

min

x

bAx

ts

xcT

nmnm cbA ,,

Standard Form LP

• In a standard form LP, the coefficient

matrix A is assumed to have full row rank

– That is, the rows of A are linearly independent

• You cannot find a linear combination of rows of A

that is equal to zero vector

• If A does not have full row rank, one can

easily reduce it to a full row rank matrix by

elementary row operations

Standard Form LP

• Used while implementing Simplex Method to

solve an LP

• A general form LP can be converted to a

standard form LP

– How?

• Objective function? (max -> min)

– Multiply by -1

• Inequalities?

– Add slack or excess (surplus) variables

• Variables unrestricted in sign? (xi - free)

0,

ii

iii

xx

xxx

Example

freexx

xx

xx

x

ts

xx

21

21

21

1

21

,0

1823

122

4

.

53max

• Convert this LP to a standard form LP

Example

0,,,,

18223

122

4

.

553min

11221

1221

221

11

221

esxxx

exxx

xxx

sx

ts

xxx

Some Definitions

• An LP is feasible if the feasible region of the

problem is not empty, or equivalently, if there is a

vector x’ which satisfies all the constraints

– x’ is called a feasible solution if it satisfies all the

constraints

– x’ is an optimal solution if it is a feasible solution and

minimizes the objective function value among all

feasible solutions

• An LP is called unbounded if there exists a

sequence of feasible solutions {xk} such that

kT xc

Some Definitions

• If an LP is feasible but not unbounded, then it

has an optimal solution

• In summary, there are three alternatives for an

LP

– It can be feasible and has an optimal solution

– It can be unbounded

– It can be infeasible

• Feasible region is empty

Duality

• Associated with every (primal) LP, there is

another LP called dual LP

• Main idea behind duality:

– To find a bound on the optimal solution of the

primal problem

• Dual of a dual LP is the primal LP

Duality

• Consider the following LP

0,,

33

14

..

34max

321

321

21

321

xxx

xxx

xx

ts

xxx

Duality

• Any feasible solution to this LP provides a

lower bound on the optimal objective

function value

– (x1,x2,x3) = (1,0,0)

• Optimal objective function value ≥ 4

– (x1,x2,x3) = (0,0,3)

• Optimal objective function value ≥ 9

• How good is this bound?

• Is it close to the optimal value?

Duality

• To answer these questions, we have to

find an upper bound

• In the given example

– Multiply the first constraint by 2 and multiply

the second constraint by 3 and sum those two

inequalities

Duality

113511

)3by multiply (33

)2by multiply (14

321

321

21

xxx

xxx

xx

• Optimal objective function value is not greater

than 11

11351134 321321 xxxxxx

Duality

• What is the best upper bound that can be found

using this method?

2132221121

2321

121

3)()4()3(

)by multiply (33

)by multiply (14

yyxyxyyxyy

yxxx

yxx

• To use this inequality for finding an upper bound,

coefficient of xi’s should be at least as large as

the corresponding coefficient in the objective

function

Duality

• And, we want to minimize the right hand side of

the inequality to find the best upper bound

0,

3

14

43

..

3min

21

2

21

21

21

yy

y

yy

yy

ts

yy

Dual LP

Dual of a Standard Form LP

0

..

min

x

bAx

ts

xcT

freey

cyA

ts

yb

T

T

..

max

Primal LP Dual LP

Primal and Dual Summary

Duality

• We can interpret the optimal dual prices yi

as the marginal cost of changing the right

hand side of the ith equation

– Why?

Economic Interpretation

• The dual prices, or shadow prices allow us to put a value on resources

• Alternatively, they allow us to consider the sensitivity of the optimal solution value to changes in the input

• By examining the dual variable for each constraint, we can determine the value of an extra unit of the corresponding resource

• We can then determine the maximum amount we would be willing to pay to have a unit of that resource

Weak Duality Theorem

• Let x be any feasible solution to the primal

LP and y be any feasible solution to the

dual LP. Then,

– cTx ≥ bTy

• Proof?

– x ≥ 0 and c - ATy ≥ 0

– Check the inner product of these two vectors

• cTx – bTy is called the duality gap

Consequences of Weak Duality

• If the primal LP is unbounded, then the

dual LP must be infeasible

• If the dual LP is unbounded, then the

primal LP must be infeasible

• If x is feasible for the primal LP, y is

feasible for the dual LP, and cTx = bTy,

then x must be optimal for the primal LP

and y must be optimal for the dual LP

Strong Duality Theorem

• If both the primal LP and the dual LP have

feasible solutions, then they both have

optimal solutions and for any primal

optimal solution x and dual optimal

solution y, we have cTx = bTy

Strong Duality Theorem: Optimality

Conditions

• x ∈ Rn is an optimal solution for the primal

LP if and only if

– x is primal feasible: Ax = b, x ≥ 0

– and there exists y ∈ Rm such that y is dual

feasible: ATy ≤ c

– and there is no duality gap: cTx = bTy

Complementary Slackness

• x ∈ Rn is an optimal solution for the primal

LP if and only if

– x is primal feasible: Ax = b, x ≥ 0

– and there exists y ∈ Rm such that y is dual

feasible: s:= c - ATy ≥ 0

– and we have xisi = 0 for each i ∈ {1,2,…,n}

(This is equivalent to the last condition of the

optimality conditions due to strong duality)

Example


• You are given the information that x2 and x3 are

positive in the optimal solution. Use the

complementary slackness conditions to find the

optimal dual solution.

Solving an LP

• Simplex method

– Moves on the boundary of the feasible region

from one corner (extreme) point to another

one

• Interior point methods

– Moves along a path in the interior of the

feasible region

Some More Definitions

• A polyhedron is a set of the form

for some matrix B, and vector d

• Let a be a vector of dimension n and b is a real

number, then is called a

halfspace

• A polyhedron is an intersection of halfspaces

}{ dBxx n

}{ bxax Tn

Some More Definitions

• Let P a polyhedron in Rn. A vector x ∈ P is

an extreme point if it cannot written as a

convex combination of two other points in

P

– That is, there does not exist y, z ∈ P and λ ∈

(0,1) such that x = λy+(1- λ)z

Simplex Method Basics

• Consider a standard form LP

where A is a matrix of dimension m×n, b is a

vector of dimension m, and c is a vector of

dimension n

• Recall that A has full row rank, i.e., m ≤ n

0

..

min

x

bAx

ts

xcT


• Let S denote the feasible region of the standard

form LP

• Note that S is also a polyhedron

• Partition the columns of matrix A into two sets of

B and N such that B is of dimension m×m and

has full rank (i.e., B-1 exists)

– A = [ B N ]

• B is called a basis matrix

}0,{ xbAxxS n

Simplex Methods Basics • Partition the decision vector x into two sets xB

and xN such that the variables in xB (or xN)

correspond to the columns in B (or N)

– That is,

• Then, we have B xB + N xN = b

• Setting xN= 0, we have xB = B-1b

• is called basic solution

N

B

x

xx

0

1bB

x

xx

N

B


• xB is called basic variables, and xN is

called nonbasic variables

• If a basic solution is feasible, i.e., B-1b ≥ 0,

then it is called a basic feasible solution

Simplex Method Basics • x ∈ S is an extreme point of S if and only if x is a

basic feasible solution

• Two extreme points are adjacent if they differ in only one basic variable

• While moving from one extreme point to another one, the new basic variable is called the entering variable, and the new nonbasic variable is called the leaving variable

Basic Theorem of LP

• Basic Theorem of LP: Consider a

standard form LP. If the feasible

region (S) has at least one extreme

point and there exists an optimal

solution, then there exists an optimal

solution that is an extreme point


• Simplex method starts from an extreme point and moves to an adjacent extreme point if the objective function value improves

• Since there are only a finite number of extreme points, the method terminates at an optimal solution after a finite number of iterations or detects that the problem is infeasible or unbounded

Simplex Method Illustration

Simplex Algorithm Basics

• At an extreme point, all the nonbasic variables are zero and the basic variables are greater than or equal to zero.

• To move from one extreme point to another one, we increase the value of a nonbasic variable (this is the entering variable) and look at the changes in the values of the basic variables to determine the leaving variable


• Let B be a basis, then

N

T

B

T

N

T

B

N

T

NN

T

B

NBNB

N

T

NB

T

B

T

N

B

xNBccbBc

xcNxBbBcz

NxBbBxbNxBxAx

xcxcxcz

x

xx

NBA

)(

)(

11

1

1

11



N

T

B

T

N

T

B

N

T

NN

T

B

NBNB

N

T

NB

T

B

T

N

B

xNBccbBc

xcNxBbBcz

NxBbBxbNxBxAx

xcxcxcz

x

xx

NBA

)(

)(

11

1

1

11



N

T

B

T

N

T

B

N

T

NN

T

B

NBNB

N

T

NB

T

B

T

N

B

xNBccbBc

xcNxBbBcz

NxBbBxbNxBxAx

xcxcxcz

x

xx

NBA

)(

)(

11

1

1

11



N

T

B

T

N

T

B

N

T

NN

T

B

NBNB

N

T

NB

T

B

T

N

B

xNBccbBc

xcNxBbBcz

NxBbBxbNxBxAx

xcxcxcz

x

xx

NBA

)(

)(

11

1

1

11


• Let’s assume that this basic solution is feasible,

i.e., B-1b ≥ 0, then

N

T

B

T

N

T

B

NB

xNBccbBcz

NxBbBx

)( 11

11


• What happens when we increase the

value of a nonbasic variable, say xj?

• Let aj be the column of A that corresponds

to xj

• Let’s look at the changes in the values of

the basic variables and the objective

function


j

jT

Bj

T

B

j

j

B

xaBccbBcz

xaBbBx

)( 11

11

• If , then increasing the value of

xj improves the objective function value.

Otherwise, there is no point of increasing the

value of xj

– Observation: Choose a variable with

– Also called the entering variable

• is called the reduced cost

01 jT

Bj aBcc

01 jT

Bj aBcc

jT

Bj aBcc 1


• If the ith element of B-1aj is positive, i.e.,

[B-1aj ]i > 0, then the value of basic variable xi decreases as we increase the value of xj

• If B-1aj ≤ 0 (assuming ), then the LP is unbounded – WHY?

• Among the basic variables that decrease as we increase the value of the entering variable, the one that reaches to zero first (also called leaving variable) leaves the basis and becomes a nonbasic variable – Determined based on the ratio test

01 jT

Bj aBcc

Simplex Method Outline • Start with a basic feasible solution

• Determine the entering variable (a nonbasic variable with a negative reduced cost)

– A popular way is to choose the nonbasic variable with the most negative reduced cost

– If there is no entering variable, then the current solution is optimal

• Determine the leaving variable

– If no leaving variable, problem is unbounded

• Form the new basis, and go to Step 2.

Simplex Method in Tableau Format

Simplex Method: Example


0,

722

205

42

.

27min

21

21

21

21

21

xx

xx

xx

xx

ts

xx


• First, convert it to a standard form LP

0,,,,

722

205

42

.

27min

54321

521

421

321

21

xxxxx

xxx

xxx

xxx

ts

xx


• The initial tableau is as follows:


• Choose x1, x3 and x4 as the basic variables in

the initial basis B

• The corresponding (basic feasible) solution is

as follows:


• The basic feasible solution is

0

25

215

0

27

5

4

3

2

1

x

x

x

x

x

x


• Is this solution optimal?

• No, we can improve the objective function value by increasing the value of x5

– The reduced cost of x5 is -7/2

• Choose x5 as the entering variable

• Perform the ratio test to determine the leaving variable:

– There is only one candidate (x4 leaves the basis)


• The new basic variables are x1, x3 and x5

• And the new tableau is



• No, we can improve the objective function value

by increasing the value of x2

– The reduced cost of x2 is -5/3

• Choose x2 as the entering variable

• Perform the ratio test to determine the leaving

variable:

basis theleaves11

40}

5/1

4,

5/11

8min{ 3x


• The new basic variables are x1, x2 and x5

• And the new tableau is



– Yes

• The optimal solution is as follows:

– x1=36/11, x2=40/11, x3=0, x4=0, x5=75/11

• Optimal objective function value is -332/11

• Note that the objective function value improved

at each iteration

Graphical Illustration

Optimization Software

• Excel Solver

• CPLEX

• GAMS

• Matlab

• …

Example using Excel Solver

• Consider the following standard form LP

0,,,,

722

205

42

.

27min

54321

521

421

321

21

xxxxx

xxx

xxx

xxx

ts

xx

• Solve it using Excel Solver

Another Example • A bond portfolio manager has $100,000 to allocate to two

different bonds; one corporate and one government bond. The corporate bond has a yield of 4%, a maturity of 3 years and an A rating from Moody's that is translated into a numerical rating of 2 for computational purposes. In contrast, the government bond has a yield of 3%, a maturity of 4 years and rating of Aaa with the corresponding numerical rating of 1 (lower numerical ratings correspond to higher quality bonds). The portfolio manager would like to allocate her funds so that the average rating for the portfolio is no worse than Aa (numerical equivalent 1.5) and average maturity of the portfolio is at most 3.6 years. Any amount not invested in the two bonds will be kept in a cash account that is assumed to earn no interest for simplicity and does not contribute to the average rating or maturity computations. How should the manager allocate her funds between these two bonds to achieve her objective of maximizing the yield from this investment?

Another Example: Formulation

• Let x1 and x2 denote the amount of funds (in

thousands of dollars) allocated to the corporate

and government bonds, respectively

0,

6.343

5.12

100

.

34max

21

21

21

21

21

21

21

xx

xx

xx

xx

xx

xx

ts

xx

Linear?

Another Example: Formulation

0,

06.04.0

05.05.0

100

.

34max

21

12

21

21

21

xx

xx

xx

xx

ts

xx

Another Example: Solution

• Exercise: Solve the model in the previous

slide using the simplex method

• Solution found by Excel Solver

– x1 = 50, x2 = 50, z = 350

Documents

LP Introduction