Basic Feasible Solutions: Recap

MS&E 211

WILL FOLLOW A CELEBRATED INTELLECTUAL TEACHING TRADITION

TEN STEPS TOWARDS UNDERSTANDING BASIC FEASIBLE SOLUTIONS

THESE SLIDES: ONLY INTUITIONDETAILS ALREADY COVERED

Step 0: Notation

• Assume an LP in the following formMaximize cTxSubject to:

Ax ≤ b x ≥ 0

• N Variables, M constraints• U = Set of all feasible solutions

Step 1: Convex Sets and Convex Combinations

• Convex set: If two points belong to the set, then any point on the line segment joining them also belongs to the set

• Convex combination: weighted average of two or more points, such that the sum of weights is 1 and all weights are non-negative– Simple example: average

Convex Combination

• Given points x1, x2, …, xK , in N dimensions• For any scalars a1, a2, …, aK such that– Each ai is non-negative

– a1 + a2 + … + aK = 1

a1x1 + a2x2 + … + aKxK is a convex combination of x1, x2, …, xK

Example: Convex combination of two points

Exercise

• If w is a convex combination of x and y, and z is a convex combination of x and w, then must z be a convex combination of x and y?

Example: Convex combination of three points

Exercise

Is the set of all convex combinations of N points always convex?

Exercise

Can any convex set be represented as a convex combination of N points, for finite N?

Step 2: Half spaces are convex

• aTx ≤ b, aTy ≤ b

• Choose z = (x+y)/2• Then, we must have aTz ≤ b

Step 3: Intersection of convex sets is convex

• Two convex sets S, T• Suppose x, y both belong to S ∩T• z = (x+y)/2

(1) z belongs to S since S is convex(2) z belongs to T since T is convex

• Hence, z belongs to S ∩ T• Implication: S∩T is convex

Step 4: Feasible solutions to LP

• The set U of feasible solutions to a Linear Program is convex

• Proof: U is the intersection of Half Spaces

Exercise

• Must every convex set be bounded?

Step 5: Basic Feasible Solution

• x is a basic feasible solution to a LP, if

(1) x is a feasible solution

(2) There do not exist two other feasible solutions y, z such that x = (y+z)/2

ALSO known as vertex solution, extreme point solution, corner-point solution

Step 6: BFS and Bounded Polytopes

• If U is bounded, then any point in U is the convex combination of its basic feasible solutions

Use b1, b2, …, bK to denote the K basic feasible solutions

Step 7: BFS and Optimality for Bounded Polytopes

• If U is bounded and non-empty, then there exists an optimal solution that is also basic feasible

Step 8: BFS and Optimality for General LPs

• If an LP has a basic feasible solution and an optimum solution, then there exists an optimal solution that is also basic feasible

• Will call these solutions “Vertex Optimal” or “Optimum Basic Feasible”

Step 9: Two Implications

(1) SOLUTION OF LPs: The Simplex method which walks from BFS to BFS is correct for bounded polytopes

(2) STRUCTURAL CONSEQUENCES: The Vertex Optimal Solutions often have very interesting properties. Example: For the matching LP, every vertex optimal solution is integral

In Practice

• Most LP Solvers return an optimum basic feasible solution, when one exists.– Either, they use Simplex– Or, they transform the solution that they do find

to a basic feasible solution• Hence, when we solve a problem using Excel

we get an optimum basic feasible solution, when one exists.

Step 10: A useful property

• At every basic feasible solution, at least N constraints are tight

• More specifically: Exactly N Linearly Independent Constraints are tight

• Basic solution: Not necessarily feasible, but exactly N Linearly Independent Constraints are tight

Exercise

You are given a LP with three decision variables where some of the constraints are

0 ≤ x ≤ 10 ≤ y ≤ 10 ≤ z ≤ 1

Is it possible to add other constraints and an objective function such that the LP has an optimum solution but not an optimum basic feasible solution?

LP in standard form

Minimize cTxSubject to:

Ax = b x ≥ 0

• N Variables, M constraints, assume that the M constraints are Linearly Independent

• At any BFS, at most M variables are strictly positive

Recap

• A feasible solution is basic feasible if it is not the average of two other feasible solutions

• If the feasibility region U for a LP is bounded and non-empty, then there exists an optimal solution that is also basic feasible

• If an LP has a basic feasible solution and an optimum solution, then there exists an optimal solution that is also basic feasible

• Commercial LP solvers return an optimum basic feasible solution by default, where one exists

• If there are N decision variables, then a basic feasible solution has N linearly independent constraints tight

LP Example: Transportation

32

Retailer 1 Retailer 2 Retailer 3 Retailer 4 SUPPLY

Warehouse 1 12 (c11) 13 4 6 500 (s1)

Warehouse 2 6 4 10 11 700 (s2)

Warehouse 3 10 9 12 14 (c34) 800 (s3)

DEMAND 400 (d1) 900 (d2) 200 (d3) 500 (d4)

jix

jdx

isx

xc

ij

ji

ij

ij

ij

i jijij

,,0

4,3,2,1 ,

3,2,1,s.t.

min

3

1

4

1

3

1

4

1

1

2

3

1

2

3

4

x11

x34

Abstract Model

LP Example: Transportation, N warehouses, M retailers

33

At most M + N non-zero variables at a BFS

Can we get to M + N -1?

1

2

N

1

2

3

M

x11

xNM

LP Example: Transportation, N warehouses, M retailers

34

Demand = -si on the left, dj on the rightEdges directed from left to rightCosts cij

Capacities 1

MIN COST FLOW!

Hence, BFSs are going to be integral if demands and supplies are integral

1

2

N

1

2

3

M

x11

xNM

LP Example: Transportation, N warehouses, M retailers

35

MIN COST FLOW!

No cycles at BFS (else can send flow in both directions)

=> At most M+N-1 non-zero values (the maximum number of edges in a an acyclic graph on M+N vertices)

Demand = -si on the left, dj on the rightEdges directed from left to rightCosts cij

Capacities 1

1

2

N

1

2

3

M

x11

xNM

Exercise

• What if the demands and supplies don’t match up? For example, supply is bigger than the demand?

Exercise

• What if the demands and supplies don’t match up? For example, supply is bigger than the demand?

• Solution: Add a fake nodewith the residual demand,and add 0 cost, infinitecapacity edges from all supplynodes to this fake node

1

2

N

1

2

3

M

x11

xNM

Exercise

• What if there is also a cost to disposal of excess supply?

Exercise

• What if there is also a cost to disposal of excess supply?

• Solution: Add a cost on theedges from the supply nodesto the fake node

• Same idea works for excessdemand

1

2

N

1

2

3

M

x11

xNM

THE SIMPLEX METHOD

Basic and Basic Feasible Solution

41

In the LP standard form, select m linearly independent columns, denoted by the variable index set B, from A. Solve

AB xB = b

for the dimension-m vector xB . By setting the variables, xN , of x corresponding to the remaining columns of A equal to zero, we obtain a solution x such that Ax = b.

Then, x is said to be a basic solution to (LP) with respect to the basic variable set B. The variables in xB are called basic variables, those in xN are nonbasic variables, and AB is called a basis.

If a basic solution xB ≥ 0, then x is called a basic feasible solution, or BFS. Note that AB and xB follow the same index order in B.Two BFS are adjacent if they differ by exactly one basic variable.

Simplex MethodGeorge B. Dantzig’s Simplex Method for linear programming stands as one of the most significant algorithmic achievements of the 20th century. It is now over 60 years old and still going strong.

The basic idea of the simplex method to confine the search to corner points of the feasible region (of which there are only finitely many) in a most intelligent way, so the objective always improves

The key for the simplex method is to make computers see corner points; and the key for interior-point methods is to stay in the interior of the feasible region.

42

x1

x2

From Geometry to Algebra

•How to make computer recognize a corner point? BFS

•How to make computer terminate and declare optimality?

•How to make computer identify a better neighboring corner?

43

Feasible Directions at a BFS and Optimality Test

44

Recall at a BFS: AB xB +ANxN = b, and xB > 0 and xN = 0 .

Thus the feasible directions, d, are the ones to satisfyAB dB +ANdN = 0, dN ≥ 0.

For the BFS to be optimal, any feasible direction must be an ascent direction, that is,

cTd= cTB dB + cT

NdN ≥ 0.

From dB = -(AB )-1ANdN, we must have for all dN ≥ 0,

cTd= -cTB (AB )-1ANdN +cT

NdN =(cTN - cT

B (AB )-1AN) dN ≥ 0

Thus, (cTN - cT

B (AB )-1AN) ≥ 0 is necessary and sufficient. It is called the reduced cost vector for nonbasic variables.

Computing the Reduced Cost Vector

45

We compute shadow prices, yT = cT

B (AB )-1 , or yT AB = cT

B, by solving a system of linear equations.

Then we compute rT=cT-yTA, where rN is the reduced cost vector for nonbasic variables (and rB=0 always).

If one of rN is negative, then an improving feasible direction is found by increasing the corresponding nonbasic variable value

Increase along this direction till one of the basic variables becomes 0 and hence non-basic

We are left with

The process will always converge and produce an optimal solution if one exists (special care for unbounded optimum and when two basic variables become 0 at the same time)

T

Summary

• The theory of Basic Feasible Solutions leads to a solution method

• The Simplex algorithm is one of the most influential and practical algorithms of all time

• However, we will not test or assign problems on the Simplex method in this class (a testament to the fact that this method has been so successful that we can use it as a basic technology)