Upload
ladnaks
View
314
Download
0
Tags:
Embed Size (px)
Citation preview
Introduction Geometry Simplex Duality
VU Algorithmics
Part V: Linear Programming
Markus Leitner, Gunther Raidl,Mario Ruthmair
Algorithms and Data Structures GroupInstitute of Computer Graphics and Algorithms
Vienna University of Technology
WS 2011/12
VU Algorithmics M. Leitner 1
Introduction Geometry Simplex Duality
Topics of this part
Introduction to Linear Programming
Geometry of Linear Programming
The Simplex Method
Duality Theory
VU Algorithmics M. Leitner 2
Introduction Geometry Simplex Duality
Introduction
VU Algorithmics M. Leitner 3
Introduction Geometry Simplex Duality
Literature for this part
1 D. Bertsimas and J. N. Tsitsiklis, Introduction to LinearOptimization, Athena Scientific, 1997.
2 6.251J/15.081J Introduction to Mathematical Programming - Fall2009, MIT OpenCourseWare (http://ocw.mit.edu).
3 G. Nemhauser and L. A. Wolsey, Integer and CombinatorialOptimization, Wiley, 1999.
4 A. Schrijver, Theory of Linear and Integer Programming, Wiley,1986.
VU Algorithmics M. Leitner 4
Introduction Geometry Simplex Duality
What is Linear Optimization / Linear Programming?
Definition 1 (Linear Programming)
Linear programming is the problem of minimizing a linear cost functionsubject to linear equality and inequality constraints.
Example 2 (Exemplary Linear Program)
minimize 2x1 − x2 + 4x3
subject to x1 + x2 + x4 ≤ 2
3x2 − x3 = 5
x3 + x4 ≥ 3
x1 ≥ 0
x3 ≤ 0
VU Algorithmics M. Leitner 5
Introduction Geometry Simplex Duality
General Linear Programming Problem
In the general case we are given a
cost vector c = (c1, . . . , cn)
and seek to minimize a
linear cost function c′x =∑n
i=1 cixi
over all n-dimensional vectors x = (x1, . . . , xn), subject to a set of
linear equality and inequality constraints.
Let
M1, M2, and M3 be finite index sets for each of which we are givenan n-dimensional vector ai and a scalar bi , used to form the i-thconstraint.
N1 and N2 be subsets of {1, . . . , n} indicating which variables xj areconstrained to be nonnegative or nonpositive, respectively.
VU Algorithmics M. Leitner 6
Introduction Geometry Simplex Duality
General Linear Programming Problem
min c′x
s.t. ai′x ≥ bi ∀i ∈ M1
ai′x ≤ bi ∀i ∈ M2
ai′x = bi ∀i ∈ M3
xj ≥ 0 ∀j ∈ N1
xj ≤ 0 ∀j ∈ N2
decision variables x1, . . . , xn
objective (cost) function c′x
free (unrestricted) variables xj , j /∈ N1 ∪ N2
VU Algorithmics M. Leitner 7
Introduction Geometry Simplex Duality
Further Notations and Definitions
vector x satisfying all constraints is called a feasible solution orfeasible vector
set of feasible solutions is called feasible set or feasible region
a feasible solution x∗ that minimizes the objective function (x∗ ≤ xfor all feasible solutions x) is called an optimal feasible solution oroptimal solution;value c′x is then called optimal cost
VU Algorithmics M. Leitner 8
Introduction Geometry Simplex Duality
Transformation Rules
1 max c′x⇔ min c′x
2 ai′x = bi ⇔ ai′x ≤ bi ∧ ai′x ≥ bi
3 ai′x ≤ bi ⇔ −ai′x ≥ −bi
4 xj ≥ 0 and xj ≤ 0 are special cases of ai′x ≥ bi
(j-th unit vector ai, bi = 0)
⇒ feasible set of a linear programming problem can be expressedexclusively in terms of inequality constraints of the form ai′x ≥ bi .
Suppose there are
m constraints (indexed by i = 1, . . . ,m)
A is a m × n matrix with row vectors ai′, i = 1, . . . ,m
b = (b1, . . . , bm)
VU Algorithmics M. Leitner 9
Introduction Geometry Simplex Duality
Definition 3 (Linear Program in General Form (Compact))
min c′x
s.t. Ax ≥ b
x ∈ Rn
Definition 4 (Feasible Set)
P = {x ∈ Rn : Ax ≥ b}
VU Algorithmics M. Leitner 10
Introduction Geometry Simplex Duality
Where do Linear Optimization Programs arise?
Real world problems
may have a (straightforward) description as LP or
may be approximated as LPs
modeling is an important task!
Integer Linear Programs ⇒ last part of “Algorithmics”,mathematical programming (summer term)
Transportations and Logistics
Telecommunications
Scheduling
Manufacturing
Engineering
Portfolio optimization
. . .
VU Algorithmics M. Leitner 11
Introduction Geometry Simplex Duality
Exemplary Transportation Problem (Data)
Given
m plants
n warehouses
si supply of ith plant, i = 1, . . . ,m
dj demand of jth warehouse, j = 1, . . . , n
cij transportation costs per unit from plant i to warehouse j∑mi=1 si =
∑nj=1 dj (for simplicity)
Objective
find a minimum cost transportation plan such that
all demands are satisfied andand all supplies are respected
VU Algorithmics M. Leitner 12
Introduction Geometry Simplex Duality
Exemplary Transportation Problem (Formulation)
Decision variables
xij . . . number of units to send from supply i to plant j
Formulation (Linear Program)
minm∑i=1
n∑j=1
cijxij
s.t.m∑i=1
xij = dj j = 1, . . . , n
n∑j=1
xij = si i = 1, . . . ,m
xij ≥ 0
VU Algorithmics M. Leitner 13
Introduction Geometry Simplex Duality
Exemplary (Nurse) Scheduling Problem (Data)
Given
planning horizon of seven days
nurses work five days in a row
dj : number of nurses needed on day j , j = 1, . . . , 7
Objective
identify the minimum number of nurses to be hired
VU Algorithmics M. Leitner 14
Introduction Geometry Simplex Duality
Exemplary Scheduling Problem (Formulation)Decision variables
xj . . . number of nurses starting to work on day j
Formulation (Linear Program)
min x1 +x2 +x3 +x4 +x5 +x6 +x7x1 +x4 +x5 + x6 +x7 ≥ d1
x1 +x2 +x5 + x6 +x7 ≥ d2
x1 +x2 +x3 + x6 +x7 ≥ d3
x1 +x2 +x3 +x4 +x7 ≥ d4
x1 +x2 +x3 +x4 +x5 ≥ d5
+x2 +x3 +x4 +x5 +x6 ≥ d6
+x3 +x4 +x5 +x6 +x7 ≥ d7
xj ≥ 0 j = 1, . . . , 7
VU Algorithmics M. Leitner 15
Introduction Geometry Simplex Duality
Guidelines
How to formulate
1 define your decision variables
2 write constraints and objective function
Good LP formulations often have
a small number of variables and constraints
sparse constraint matrix
Reminder: Models must be linear, e.g. you are not allowed to multiplytwo variables!
VU Algorithmics M. Leitner 16
Introduction Geometry Simplex Duality
Standard form problems
Definition 5 (Standard Form)
A linear programming problem of the form
min c′x
s.t. Ax = b
x ≥ 0
is said to be in standard form.
Interpretation
x ∈ Rn, Ai is ith column of A
then Ax = b can be written as∑n
i=1Aixi = b
Ai are resource vectors that should synthesize b by using anonnegative amount xi of each resource
ci is the unit cost of each resource
VU Algorithmics M. Leitner 17
Introduction Geometry Simplex Duality
Reduction to standard form
obviously, standard form is special case of general form
we will see that general form can be transformed into standard form
Note
usually, the general form Ax ≥ b is used to develop the theory oflinear programming
when talking about algorithms, e.g. simplex method, standard formAx = b, x ≥ 0, will be used (computationally more convenient)
VU Algorithmics M. Leitner 18
Introduction Geometry Simplex Duality
Reduction rules
Elimination of free variables
Assume xj is an an unrestricted variable in a general form problem
xj ⇔ x+j − x−j with x+
j , x−j ≥ 0
Elimination of inequality constraints
Given an inequality∑n
j=1 aijxj ≤ bi
introduce a slack variable si ≥ 0∑nj=1 aijxj + si = bi
Given an inequality∑n
j=1 aijxj ≥ bi
introduce a surplus variable si ≥ 0∑nj=1 aijxj − si = bi
VU Algorithmics M. Leitner 19
Introduction Geometry Simplex Duality
Reduction to standard form (Example)
The problem
min 2x1 + 4x2
s.t. x1 + x2 ≥ 3
3x1 + 2x2 = 14
x1 ≥ 0
is equivalent to the standard form problem
min 2x1 + 4x+2 − 4x−2
s.t. x1 + x+2 − x−2 − x3 = 3
3x1 + 2x+2 − 2x−2 = 14
x1, x+2 , x
−2 , x3 ≥ 0
VU Algorithmics M. Leitner 20
Introduction Geometry Simplex Duality
Graphical Representation andSolution of LPs
VU Algorithmics M. Leitner 21
Introduction Geometry Simplex Duality
Graphical Representation of LPs
Motivation
provide useful geometric insights into the nature of linearprogramming problems
LPs with two variables can (easily) be solved geometrically
HowTo (Minimization Problem)
1 determine feasible region
2 move line corresponding to objective function in direction ofdecreasing objective value
3 stop if no further moving is possible without leaving the feasibleregion
4 optimal solutions are then given by all feasible points of “objectivefunction line”
VU Algorithmics M. Leitner 22
Introduction Geometry Simplex Duality
Graphical Representation of LPs (Example 1)
min − x1 − x2
s.t. x1 + 2x2 ≤ 3
2x1 + x2 ≤ 3
x1, x2 ≥ 0
line z = c′x = −x1 − x2is perpendicular to c
z = −2 is optimal value
Figure from Bertsimas, Tsitsiklis, 1997
VU Algorithmics M. Leitner 23
Introduction Geometry Simplex Duality
Graphical Representation of LPs (Example 2)
−x1 + x2 ≤ 1 and x1, x2 ≥ 0
Figure from Bertsimas, Tsitsiklis, 1997
VU Algorithmics M. Leitner 24
Introduction Geometry Simplex Duality
Optimal Solutions of a Linear Program
For a Linear Program
min c′x
s.t. Ax ≤ b
with feasible set P = {x | Ax ≤ b}
the following possibilities exist
P = ∅ ⇒ no feasible solution exists
optimal cost is −∞, i.e. inf{c′x | x ∈ P} does not exist, and henceno feasible solution is optimal (LP is unbounded)
a unique optimal solution exists
multiple optimal solutions exist; set of optimal solutions may bebounded or unbounded
VU Algorithmics M. Leitner 25
Introduction Geometry Simplex Duality
Oil Refinery (Modeling)
create different end products from raw oil
heavy oil (H)medium oil (M)light oil (L)
different crack processes produce different quantities
crack process 1: 2H, 2M, 1L costs: 3 EURcrack process 2: 1H, 2M, 4L costs: 5 EUR
required quantities
3H, 5M, 4L
Objective: minimize costs
VU Algorithmics M. Leitner 26
Introduction Geometry Simplex Duality
Oil Refinery (Variables)
Variables
x1: level of production of crack process 1
x2: level of production of crack process 2
Interpretation of x1 = 2.5
process 1 is applied to 2.5 units raw oil
costs 2.5 ∗ 3 EUR
result 2.5 ∗ 2 H, 2.5 ∗ 2 M and 2.5 ∗ 1 L
Each non-negative vector (x1, x2) ∈ R2 describes a certain level ofproduction of both crack processes.
VU Algorithmics M. Leitner 27
Introduction Geometry Simplex Duality
Oil Refinery (Constraints)
at least 3 units H2x1 + x2 ≥ 3
at least 5 units M2x1 + 2x2 ≥ 5
at least 4 units Lx1 + 4x2 ≥ 4
total costsz = 3x1 + 5x2
VU Algorithmics M. Leitner 28
Introduction Geometry Simplex Duality
Oil Refinery (Linear Program)
min 3x1 + 5x2
s.t. 2x1 + x2 ≥ 3
2x1 + 2x2 ≥ 5
x1 + 4x2 ≥ 4
x1 ≥ 0
x2 ≥ 0
VU Algorithmics M. Leitner 29
Introduction Geometry Simplex Duality
Oil Refinery (Graphical Solution)
0
C1 C2 C3
C4
C5
x2
x1����������������������������������������������
����������������������������������������������
������������������������������������������������������������������������
����
����
����
����
����
����
����
��
����
��
��
����
��
����������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������
����������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������
������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������
������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������
������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������
������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������
5
1 2 3 4 5 6
1
2
3
4
feasible region
VU Algorithmics M. Leitner 30
Introduction Geometry Simplex Duality
Oil Refinery (Graphical Solution)
������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������
������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������
0
C1 C3 C3
C5
x2
x1
C4
����������������������������������������������
����������������������������������������������
������������������������������������������������������������������������
����
����
����
����
����
����
����
��
����
��
��
����
��
����������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������
����������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������
������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������
������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������
������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������
������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������
z=19 e.g.: x=(3,2)
5
1 2 3 4 5 6
1
2
3
4
VU Algorithmics M. Leitner 31
Introduction Geometry Simplex Duality
Oil Refinery (Graphical Solution)
������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������
������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������
0
C1 C2 C3
C4
C5
x2
x1����������������������������������������������
����������������������������������������������
������������������������������������������������������������������������
����
����
����
����
����
����
����
��
����
��
��
����
��
����������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������
����������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������
������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������
������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������
������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������
������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������
5
1 2 3 4 5 6
1
2
3
4
VU Algorithmics M. Leitner 32
Introduction Geometry Simplex Duality
Oil Refinery (Graphical Solution)
����������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������
����������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������
0
C1 C2 C3
C4
C5
x2
x1����������������������������������������������
����������������������������������������������
������������������������������������������������������������������������
����
����
����
����
����
����
����
��
����
��
��
����
��
����������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������
����������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������
������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������
������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������
������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������
������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������
5
1 2 3 4 5 6
1
2
3
4
z=8.5, x*=(2, 0.5)
VU Algorithmics M. Leitner 33
Introduction Geometry Simplex Duality
The Geometry of Linear Programming
VU Algorithmics M. Leitner 34
Introduction Geometry Simplex Duality
Linear Combination
Definition 6
Let x1, x2, . . . , xk ∈ R and
λ =
λ1...λk
∈ Rk .
Then
y =k∑
i=1
λixi
is called linear combination of x1, . . . , xk .
VU Algorithmics M. Leitner 35
Introduction Geometry Simplex Duality
Specific Linear Combinations
Definition 7
If
y =k∑
i=1
λixi
andλ ≥ 0∑k
i=1 λi = 1∑ki=1 λi = 1, λ ≥ 0
then y is called
conicaffineconvex
combination of x1, x2, . . . , xk .
VU Algorithmics M. Leitner 36
Introduction Geometry Simplex Duality
Sets of Linear Combinations
Definition 8
For ∅ 6= S ⊆ Rn
lin(S)cone(S)aff(S)conv(S)
is the
linearconicaffineconvex
hull of S .
VU Algorithmics M. Leitner 37
Introduction Geometry Simplex Duality
Sets of Linear Combinations
VU Algorithmics M. Leitner 38
Introduction Geometry Simplex Duality
Sets of Linear Combinations
Definition 9
IfL = lin(S)L = cone(S)L = aff(S)L = conv(S)
, then L is a
linear spaceconeaffine spaceconvex set
VU Algorithmics M. Leitner 39
Introduction Geometry Simplex Duality
Subspaces
Definition 10
L ⊆ Rn is a linear subspace of Rn ⇔there exists A ∈ Rm×n with L = {x ∈ Rn | Ax = 0}.
Definition 11
L ⊆ Rn is a affine subspace of Rn ⇔there exist A ∈ Rm×n and b ∈ Rm with L = {x ∈ Rn | Ax = b}.
VU Algorithmics M. Leitner 40
Introduction Geometry Simplex Duality
Hyperplanes, Halfspaces, and Polyhedra
Definition 12 (Polyhedron)
A polyhedron P is a set that can be described in the formP = {x ∈ Rn | Ax ≥ b} where A ∈ Rm×n and b ∈ Rm.(cmp. feasible set of LPs)
Definition 13
A set S ⊆ Rn is bounded if there exists a constant K such that theabsolute value of every component of every element of S is less thanequal to K .
Definition 14 (Polytope)
A bounded polyhedron is called polytope, i.e.
P ⊆ {x ∈ Rn | ||x|| ≤ B} for some B ≥ 0.
VU Algorithmics M. Leitner 41
Introduction Geometry Simplex Duality
Hyperplanes, Halfspaces, and Polyhedra
Definition 15
Let a be a nonzero vector in Rn and let b be a scalar
(a) the set {x ∈ Rn | a′x ≥ b} is called a halfspace
(b) the set {x ∈ Rn | a′x = b} is called a hyperplane
a hyperplane is the boundary of a corresponding halfspace
a hyperplane is an affine subspace (m = 1)
halfspaces are polyhedrons
each polyhedron is equal to the intersection of a finite number ofhalfspaces
a polyhedron P is a convex set, i.e. if x, y ∈ P, thenλx + (1− λ)y ∈ P⇒ the feasible set of each LP is a convex set
VU Algorithmics M. Leitner 42
Introduction Geometry Simplex Duality
Hyperplanes, Halfspaces, and Polyhedra
Figure from Bertsimas, Tsitsiklis, 1997
VU Algorithmics M. Leitner 43
Introduction Geometry Simplex Duality
Representation
Theorem (Minkowski and Weyl)
A polyhedron P ∈ Rn can be represented in the form
P = conv(V ) + cone(E )
with V and E being finite subsets of Rn.
Remark
polytopes can be represented as P = conv(V )
VU Algorithmics M. Leitner 44
Introduction Geometry Simplex Duality
Example
Polytope P represented as:
P =
{(x
y
)∈ R2 | x ≤ 1, y ≤ 1, x + y ≥ 1
}
VU Algorithmics M. Leitner 45
Introduction Geometry Simplex Duality
Faces
Definition 16 (Valid Inequality)
The inequality a′x ≤ a0 is called a valid inequality for a polyhedron P ifit is satisfied by all points in P.
Definition 17
Given a polyhedron P = {x ∈ Rn : Ax ≤ b},F ⊆ P is a face of P, if there exists a valid inequality a′x ≤ a0 forP, such that F = {x ∈ P : a′x = a0}.a face F is a proper face if F 6= ∅ and F 6= P.
A facet of P is a maximal (relative to inclusion) proper face of P.
A “corner point” of P is a face of P consisting of one element.
VU Algorithmics M. Leitner 46
Introduction Geometry Simplex Duality
Faces
x_1 + x_2 <=2
x_1 <=1facet:
corner point:
P
1
1
0
VU Algorithmics M. Leitner 47
Introduction Geometry Simplex Duality
Extreme Points
Definition 18 (Extreme Point)
Let P be a polyhedron. A vector x ∈ P is an extreme point of P if wecannot find two vectors y, z ∈ P, both different from x, and a scalarλ ∈ [0, 1], such that x = λy + (1− λ)z.
Figure from Bertsimas, Tsitsiklis, 1997
x is an extreme point of P
u, v, w, y, z are no extremepoints of P
VU Algorithmics M. Leitner 48
Introduction Geometry Simplex Duality
Vertices
Definition 19 (Vertex)
Let P be a polyhedron. A vector x ∈ P is a vertex of P if there existssome c such that c′x < c′y for all y satisfying y ∈ P and y 6= x.
Figure from Bertsimas, Tsitsiklis, 1997
x is a vertex of P
w is not a vertex of P
VU Algorithmics M. Leitner 49
Introduction Geometry Simplex Duality
Active ConstraintsConsider a polyhedron P ⊆ Rn defined by linear equality and inequalityconstraints
ai′x ≥ bi , i ∈ M1
ai′x ≤ bi , i ∈ M2
ai′x = bi , i ∈ M3
where M1, M2, and M3 are finite index sets, ai ∈ Rn, and bi ∈ R.
Definition 20
If a vector x∗ satisfies ai′x∗ = bi for some i ∈ M1, M2, or M3, we saythat the corresponding constraint is active or binding at x∗.
If n constraints are active at x∗, x∗ satisfies a system of n linearequations in n unknowns. This system has a unique solution if and onlyif these n equations are linearly independent.
VU Algorithmics M. Leitner 50
Introduction Geometry Simplex Duality
Theorem 21
Let x∗ ∈ Rn and I = {i | a′ix∗ = bi} be the set of indices of constraintsthat are active at x∗. Then the following are equivalent
(a) there exist n vectors in the set {ai | i ∈ I}, which are linearlyindependent
(b) the span of the vectors ai , i ∈ I , is Rn, i.e. every element of Rn canbe expressed as linear combinations of the vectors ai , i ∈ I
(c) the system of equations a′ix∗ = bi , i ∈ I , has a unique solution
VU Algorithmics M. Leitner 51
Introduction Geometry Simplex Duality
Basic Solutions
Definition 22
Consider a polyhedron P defined by linear equality and inequalityconstraints, and let x∗ be an element of Rn
(a) vector x∗ is a basic solution if
(i) all equality constraints are active(ii) out of the constraints that are active at x∗, there are n of them that
are linearly independent
(b) if x∗ is a basic solution that satisfies all of the constraints, we saythat it is a basic feasible solution (BFS)
VU Algorithmics M. Leitner 52
Introduction Geometry Simplex Duality
Active Constraints and Basic Feasible Solutions
P = {(x1, x2, x3) | x1 + x2 + x3 = 1, x1, x2, x3 ≥ 0}
Figure from Bertsimas, Tsitsiklis, 1997
3 constraints active at A,B, C , and D
2 constraints active at E ,i.e. x1 + x2 + x3 = 1 andx2 = 0
A, B, and C are basicfeasible solutions
VU Algorithmics M. Leitner 53
Introduction Geometry Simplex Duality
Extreme Point, Vertex, Basic Feasible Solution
Theorem 23
Let P be a nonempty polyhedron and let x∗ ∈ P. Then, the following areequivalent:
(a) x∗ is a vertex
(b) x∗ is an extreme point
(c) x∗ is a basic feasible solution
Corollary 24
Given a finite number of linear inequality constraints, there can only be afinite number of basic feasible solutions.
Definition 25
Two distinct basic solutions are adjacent if there are n − 1 linearlyindependent constraints that are active at both of them.
VU Algorithmics M. Leitner 54
Introduction Geometry Simplex Duality
Polyhedra in Standard Form
P = {x ∈ Rn | Ax = b, x ≥ 0}
Essential for the development of the simplex algorithm!
Theorem 26
Consider the constraints Ax = b and x ≥ 0 and assume that the m × nmatrix A has linearly independent rows. A vector x ∈ Rn is a basicfeasible solution if and only if we have Ax = b, and there exists indicesB(1), . . . ,B(m) such that:
(a) The columns AB(1), . . . ,AB(m) are linearly independent and
(b) if i 6= B(1), . . . ,B(m), then xi = 0.
VU Algorithmics M. Leitner 55
Introduction Geometry Simplex Duality
Basic Solutions for Polyhedra in Standard Form
Constructing basic solutions
1 Choose m linearly independent columns AB(1), . . . ,AB(m)
2 Let xi = 0 for all i 6= B(1), . . . ,B(m)
3 Solve the system of m equations Ax = b for the unknownsxB(1), . . . , xB(m)
if a basic solution constr. is nonneg. it is a basic feasible solutionif x is a basic solution, then xB(1), . . . , xB(m) are called basicvariables; the remaining variables are called nonbasicthe columns AB(1), . . . ,AB(m) are called the basic columns and sincethey are linearly independent, they form a basis of Rm
B(1), . . . ,B(m) are the basic indicesarranging the m basic columns next to each other, we obtain anm ×m matrix B called basis matrixBxB = b yields xB = B−1b
VU Algorithmics M. Leitner 56
Introduction Geometry Simplex Duality
Figure from Bertsimas, Tsitsiklis, 1997
standard form problem with n = 4and m = 2
A1, A2 form a basis; correspondingbasic solution is infeasible (negativevalue of x2 is needed)
A1, A3 form another basis; basicsolution is feasible
A1, A4 do not form a basis(linearly dependent)
VU Algorithmics M. Leitner 57
Introduction Geometry Simplex Duality
The Full Row Assumption on A
Theorem 27
Let P = {x | Ax = b, x ≥ 0} be a nonempty polyhedron, where A is amatrix of dimensions m × n with rows a′1, . . . , a
′m. Suppose that
rank(A) = k < m and that the rows a′i1 , . . . a′ik
are linearly independent.Consider the polyhedron
Q = {x | a′i1x = bi1 , . . . , a′ikx = bik , x ≥ 0}.
Then Q = P
Which shows that the full row rank assumption on the matrix A inTheorem 26 results in no loss of generality.
VU Algorithmics M. Leitner 58
Introduction Geometry Simplex Duality
Degeneracy
at each basic solution we have at least n active constraints
may be more than n (of course not linearly independent) →degenerate basic solution
huge impact on algorithms for solving LPs
Definition 28
A basic solution x ∈ Rn is said to be degenerate if more than n of theconstraints are active at x.
VU Algorithmics M. Leitner 59
Introduction Geometry Simplex Duality
Degeneracy
Figure from Bertsimas, Tsitsiklis, 1997
The points A and C are degenerate basic feasible solutions. The pointsB and E are nondegenerate basic feasible solutions. The point D is adegenerate basic solution.
VU Algorithmics M. Leitner 60
Introduction Geometry Simplex Duality
Degeneracy in Standard Form Problems
in standard form, the m equality constraints are active at a basicsolution
having more than n active constraints is the same as having morethan n −m variables at zero level
Definition 29
Consider the standard form polyhedron P = {x ∈ Rn | Ax = b, x ≥ 0}and let x be a basic solution. Let m be the number of rows in A. Thevector x is a degenerate basic solution if more than n −m of thecomponents of x are zero.
For a degenerate basic solution, there are typically multiple possibilitiesto choose the n −m nonbasic variables, i.e. several bases correspond tothe same basic solution.
VU Algorithmics M. Leitner 61
Introduction Geometry Simplex Duality
Degeneracy and Representation
Degeneracy is, in general, not a geometric (representation independent)property, but rather depends on the particular representation of apolyhedron.
Example
P = {(x1, x2, x3) | x1 − x2 = 0, x1 + x2 + 2x3 = 2, x1, x2, x3 ≥ 0}n = 2, m = 3, and n −m = 1
→ (1, 1, 0) is not degenerated, but (0, 0, 1) is
P = {(x1, x2, x3) | x1 − x2 = 0, x1 + x2 + 2x3 = 2, x1 ≥ 0, x3 ≥ 0}→ (0, 0, 1) not degenerated in this representation
VU Algorithmics M. Leitner 62
Introduction Geometry Simplex Duality
Existence of Extreme Points
Definition 30
A polyhedron P ⊂ Rn contains a line if there exists a vector x ∈ P and anonzero vector d ∈ Rn such that x + λd ∈ P for all scalars λ.
Theorem 31
Suppose that the polyhedron P = {x ∈ Rn | a′ix ≥ bi , i = 1, . . . ,m} isnonempty. Then, the following are equivalent:
(a) P has at least one extreme point
(b) P does not contain a line
(c) there exist n vectors out of a1, . . . , am which are linearly independent
VU Algorithmics M. Leitner 63
Introduction Geometry Simplex Duality
Existence of Extreme Points
a bounded polyhedron does not contain a line
the positive orthant {x | x ≥ 0} does not contain a line
since a polyhedron in standard form is contained in the positiveorthant, it does not contain a line either
Corollary 32
Every nonempty bounded polyhedron and every nonempty polyhedron instandard form has at least one basic feasible solution.
VU Algorithmics M. Leitner 64
Introduction Geometry Simplex Duality
Optimality of Extreme Points
Theorem 33
Consider the linear programming problem of minimizing c′x over apolyhedron P. Suppose that P has at least one extreme point and thatthere exists an optimal solution. Then there exists an optimal solutionwhich is an extreme point of P.
Figure from Bertsimas, Tsitsiklis, 1997
VU Algorithmics M. Leitner 65
Introduction Geometry Simplex Duality
Optimality of Extreme Points
Theorem 34 (stronger variant)
Consider the linear programming problem of minimizing c′x over apolyhedron P. Suppose that P has at least one extreme point. Theneither the optimal cost is equal to −∞, or there exists an extreme pointwhich is optimal.
Corollary 35
Consider the linear programming problem of minimizing c′x over anonempty polyhedron. Then, either the optimal cost is equal to −∞ orthere exists an optimal solution.
VU Algorithmics M. Leitner 66
Introduction Geometry Simplex Duality
Summary
Conclusions regarding the solutions of LPs
(a) If the feasible set is nonempty and bounded, there exists an optimalsolution. Furthermore, there exists an optimal solution which is anextreme point.
(b) If the feasible set is unbounded, there are three possibilities
(i) There exists an optimal solution which is an extreme point.(ii) There exists an optimal solution, but no optimal solution is an
extreme point. (Only possible if the feasible set has no extremepoints; This never happens in standard form problems)
(iii) The optimal cost is −∞.
(c) If the optimal cost is finite and the feasible set contains at least oneextreme point, then there are only finitely many extreme points.
(d) The number of extreme points may increase exponentially with thenumber of variables and constraints.
VU Algorithmics M. Leitner 67
Introduction Geometry Simplex Duality
The Simplex Method
VU Algorithmics M. Leitner 68
Introduction Geometry Simplex Duality
Historical Notes
simplex method was pioneered by Dantzig in 1947
comprehensive text: G. B. Dantzig, Linear Programming andExtensions, Princeton University Press, 1963
Basic Strategy
start from a vertex (BFS) of the polyhedra
may be difficult to find!
move from one BFS to another along the edges of the feasible set ina cost reducing direction
which directions / edges are possible?which one to choose?
repeat while further improvement is possible
global optimum?
VU Algorithmics M. Leitner 69
Introduction Geometry Simplex Duality
Optimality Conditions
basic strategy of many optimization algorithms: given a feasiblesolution, search its neighborhood to find a nearby feasible solutionwith lower costs (as long as it exists)
→ such algorithms terminate in a locally optimal solution which neednot to be globally optimal
LP: optimization of a convex function over a convex set!
Theorem 36 (Local Minima of Convex Functions)
Let F : Rn → R be a convex function and S ⊂ R be a convex set. Let x∗
be an element of S. Suppose x∗ is a local optimum for the problem ofminimizing f (x) over S; i.e. there exists some ε > 0 such thatf (x∗) ≤ f (x) for all x ∈ S for which ||x− x∗|| ≤ ε. Then x∗ is globallyoptimal, i.e. f (x∗) ≤ f (x) for all x ∈ S.
VU Algorithmics M. Leitner 70
Introduction Geometry Simplex Duality
Assumptions
In this chapter, we consider the standard form problem
min c′x
s.t. Ax = b
x ≥ 0
where P is the corresponding feasible set
A is a m × n matrix with linearly independent rows
Ai is the ith column of A
a′i is the ith row of A
VU Algorithmics M. Leitner 71
Introduction Geometry Simplex Duality
Feasible Directions
given a point x ∈ P we want to move in the direction of d ∈ Rn
we do only want to consider choices for d that do not immediatelytake us outside the feasible set
Definition 37 (Feasible Direction)
Let x be an element of a polyhedron P. A vector d ∈ Rn is said to be afeasible direction at x, if there exists a positive scalar θ for whichx + θd ∈ P
VU Algorithmics M. Leitner 72
Introduction Geometry Simplex Duality
Basic Direction
given BFS x, i.e. Ax = b, basis matrix B, basic variablesxB = (xB(1), . . . , xB(m)) (xB = B−1b)
move away from x to x + θd by selecting a nonbasic variable xj(initially zero) increasing it to θ
x + θd, with dj = 1 (zero for all other nonbasic variables)
A(x + θd) = b should hold (feasibility) and Ax = b holds and θ > 0,we need Ad = 0
→ 0 = Ad =∑n
i=1Aidi =∑m
i=1AB(i)dB(i) + Aj = BdB + Aj
⇒ dB = −B−1Aj
which is called jth basic direction
VU Algorithmics M. Leitner 73
Introduction Geometry Simplex Duality
Basic Direction
equality constraints are respected when moving along the basicdirection d
nonnegativity constraints?
variable xj is increasedother nonbasic variables stay zero
Other basic variables
(a) suppose x is a nondegenerate BFS, i.e. xB > 0, and thusxB + θdB ≥ 0 is θ is sufficiently small (→ feasibility is maintainedand d is a feasible direction)
(b) if x is degenerate, d is not always a feasible direction! (a basicvariable xB(i) may be zero while dB(i) is negative)
VU Algorithmics M. Leitner 74
Introduction Geometry Simplex Duality
Effects on the Cost Function
jth basic direction d: rate c′d of cost change along d is given byc′BdB + cj , where cB = (cB(1), . . . , cB(m)), which is equal to
→ cj − c′BB−1Aj
cj . . . cost per unit increase of xj−c′BB
−1Aj . . . cost of the compensating change in the basic variablesdue to Ax = b
Definition 38 (Reduced Cost)
Let x be a basic solution, let B be an associated basis matrix, and let cBbe the vector of costs of the basic variables. For each j , we define thereduced cost c j of the variable xj according to the formula
c j = cj − c′BB−1Aj
(Note: reduced costs of basic variables are zero)
VU Algorithmics M. Leitner 75
Introduction Geometry Simplex Duality
Optimality Conditions
Theorem 39
Consider a basic feasible solution x associated with a basis matrix B, andlet c be the corresponding vector of reduced costs.
(a) if c ≥ 0, then x is optimal
(b) if x is optimal and nondegenerate, then c ≥ 0
Definition 40
A basis matrix B is said to be optimal if:
(a) B−1b ≥ 0, and
(b) c′ = c′ − c′BB−1A ≥ 0′
VU Algorithmics M. Leitner 76
Introduction Geometry Simplex Duality
Development of the Simplex Method
Main Missing Part
How to move to a better BFS, when a profitable basic direction isdiscovered
assume that every BFS is nondegenerate
reduced costs have been computed and c j < 0 (jth basic direction d)
when moving along d, xj becomes positive; we say that xj (or Aj)enters the basis
VU Algorithmics M. Leitner 77
Introduction Geometry Simplex Duality
How far can we move?
∃j : c j < 0: jth basic direction allows for a cost reduction
since costs are decreasing along d we want to move as far aspossible to x + θ∗d, where
θ∗ = max{θ ≥ 0 | x + θd ∈ P}
if d ≥ 0, then x + θd ≥ 0, for all θ ≥ 0, hence θ∗ =∞ (unbounded)
if di < 0, then xi + θdi ≥ 0⇒ θ ≤ −xi/di
must be satisfied for every i with di < 0→ θ∗ = min{i|di<0}(−xi/di )
since we only need to consider basis variables→ θ∗ = min{i=1,...,m|dB(i)<0}(−xB(i)/dB(i))
θ∗ > 0, because xB(i) > 0 (nondegeneracy)
VU Algorithmics M. Leitner 78
Introduction Geometry Simplex Duality
New Basic Feasible Solution y
y = x + θ∗d
since xj = 0 and dj = 1, we have yj = θ∗ > 0
Let l be a minimizing index, i.e.
−xB(l)
dB(l)= min{i=1,...,m|dB(i)<0}
(−
xB(i)
dB(i)
)= θ∗
in particular dB(l) < 0 and xB(l) + θ∗dB(l) = 0
basic variable xB(l) becomes zero, whereas nonbasic variable xjbecomes positive
→ xj should replace xB(l) in the basis, i.e. B is replaced by new basis
matrix B by replacing AB(l) with Aj
we have y 6= x and c′y < c′x
VU Algorithmics M. Leitner 79
Introduction Geometry Simplex Duality
New Basic Feasible Solution y
Theorem 41
Let y, B, be constructed as detailed before. Then
(a) The columns AB(i), i 6= l , and Aj are linearly independent and,
therefore B is a basis matrix
(b) The vector y = x + θ∗d is a basic feasible solutions associated withthe basis matrix B.
VU Algorithmics M. Leitner 80
Introduction Geometry Simplex Duality
An Iteration of the Simplex Method
1 start with a basis (columns AB(1), . . . ,AB(m)), and associated BFS x
2 compute reduced costs c j = cj − c′BB−1Aj for all nonbasic indices j ;
if all of them are nonnegative, current solution is optimal (algorithmterminates); else choose some variable j for which c j < 0
3 compute u = −dB = B−1Aj ; if no component of u is positive, wehave θ∗ =∞ and the optimal cost is −∞ (algorithm terminates)
4 if some component of u is positive, let
θ∗ = min{i=1,...,m|ui>0}
xB(i)
ui
5 let l be such that θ∗ = xB(l)/ul ; form new basis by replacing AB(l)
with Aj (xj enters basis, xB(l) leaves basis)
VU Algorithmics M. Leitner 81
Introduction Geometry Simplex Duality
Correctness of Simplex (Nondegenerate case)
Theorem 42
Assume that the feasible set is nonempty and that every basic feasiblesolution is nondegenerate. Then, the simplex method terminates after afinite number of iterations. At termination, there are the followingpossibilities:
(a) We have an optimal basis B and an associated basic feasiblesolution which is optimal.
(b) We have found a vector d satisfying Ad = 0, d ≥ 0, and c′d < 0,and the optimal cost is −∞.
VU Algorithmics M. Leitner 82
Introduction Geometry Simplex Duality
The Simplex Method for Degenerate Problems
Suppose, we apply the same algorithm in the presence of degeneracy.Then, two new possibilities may be encountered:
(a) If current BFS x is degenerate, θ∗ can be equal to zero and thusy = x
happens if some xB(l) is equal to zero and dB(l) < 0
still, we can define a new basis B, by replacing AB(l) by Aj
(b) if θ∗ > 0, more than one of the original basis variables may becomezero at x + θ∗d; since only one variable exists the basis, we get anew BFS which is degenerate
basic changes while “staying at the same BFS” do not reduce cost
degeneracy may lead to cycling (i.e. algorithm may loop indefinitely)
VU Algorithmics M. Leitner 83
Introduction Geometry Simplex Duality
Simplex and Degeneracy (Example)
Figure from Bertsimas, Tsitsiklis, 1997
VU Algorithmics M. Leitner 84
Introduction Geometry Simplex Duality
Pivot Selection
simplex algorithm has certain degrees of freedom:selection of nonbasic variable xj with negative reduced cost
Dantzig’s rule: most negative reduced costlargest cost decrease (more computational effort!)Bland’s rule: smallest subscript (smallest index j such that c j < 0)(Devex, steepest edge, candidate lists, partial pricing,. . . )
selection of variable leaving basis (if multiple indices attain theminimum in definition of θ∗)
Bland’s rule: smallest subscript
rules for making such choices are called pivoting rules
VU Algorithmics M. Leitner 85
Introduction Geometry Simplex Duality
Full Tableau Implementation
Structure of the representation
−cTBB−1b cT − cTBB−1A
B−1b B−1A
or, in more detail
−cTB xB c1 . . . cn
xB(1) | |... B−1A1 . . . B−1An
xB(m) | |
VU Algorithmics M. Leitner 86
Introduction Geometry Simplex Duality
Full Tableau Implementation – Iteration
1 start with tableau associated with basis matrix B and BFS x
2 examine reduced costs (0th row of tableau); if they are nonnegative,current solution is optimal; else choose some j with c j < 0
3 consider vector u = B−1Aj (jth column of tableau):
if no component of u is positive, optimal cost is −∞; terminationfor each i with ui > 0 compute xB(i)/ui ; let l be index of a row withsmallest ratio; column AB(l) exists basis, Aj enters basis
4 add to each row a constant multiple of the pivot row (lth row) sothat ul (pivot element) becomes one and all other entries of thepivot column become zero
VU Algorithmics M. Leitner 87
Introduction Geometry Simplex Duality
Full Tableau Implementation (Example from [1])
After introducing slack variables, we obtain the standard form problem
VU Algorithmics M. Leitner 88
Introduction Geometry Simplex Duality
Full Tableau Implementation (Example from [1])
Basic feasible solution: x = (0, 0, 0, 20, 20, 20); accordingly B(1) = 4,B(2) = 5, B(3) = 6; B = I
→ x1 has negative reduced costs (enters basis)→ pivot column u = (1, 2, 2); form xB(i)/ui
→ minimum at i = 2 and i = 3; choose l = 2, i.e. x5 leaves the basis
VU Algorithmics M. Leitner 89
Introduction Geometry Simplex Duality
Full Tableau Implementation (Example from [1])
New (degenerate) BFS: x = (10, 0, 0, 10, 0, 0); B(1) = 4, B(2) = 1,B(3) = 6
→ choose x3 to enter basis; pivot column u = (1, 1,−1)→ u3 < 0, hence calculate xB(i)/ui for i = 1, 2 only→ tie; we chose x4 to exit the basis
VU Algorithmics M. Leitner 90
Introduction Geometry Simplex Duality
Full Tableau Implementation (Example from [1])
New (degenerate) BFS: x = (0, 0, 10, 0, 0, 10)
→ x2 enters basis, x6 exists basis
VU Algorithmics M. Leitner 91
Introduction Geometry Simplex Duality
Full Tableau Implementation (Example from [1])
New BFS: x = (4, 4, 4, 0, 0, 0)
→ optimal solution, since all reduced costs are nonnegative!
VU Algorithmics M. Leitner 92
Introduction Geometry Simplex Duality
Full Tableau Implementation (Example from [1])Feasible Set; Path A-D-B-E
Figure from Bertsimas, Tsitsiklis, 1997
VU Algorithmics M. Leitner 93
Introduction Geometry Simplex Duality
Full Tableau Implementation: Cycling (Example from [1])
Initial tableau (pivot elements are marked using “*”)
pivoting rules
nonbasic variable with most negative reduced cost c j enters the basis
out of eligible basic variables, the one with smallest index (subscript)exists the basis
VU Algorithmics M. Leitner 94
Introduction Geometry Simplex Duality
After iteration 1:
After iteration 2:
VU Algorithmics M. Leitner 95
Introduction Geometry Simplex Duality
After iteration 3:
After iteration 4:
VU Algorithmics M. Leitner 96
Introduction Geometry Simplex Duality
After iteration 5:
After iteration 6: same basis and tableau that we started with (cycling!)
VU Algorithmics M. Leitner 97
Introduction Geometry Simplex Duality
Anticycling
appropriate pivoting rules prevent cyclingLexicographic pivoting rule
choose entering variable arbitrarily (with neg. reduced cost)for each row i with ui > 0, divide ith row of tableau by ui and choselexicographic smallest row to identify variable leaving the basis
Bland’s Rulefind smallest j for which c j < 0; xj enters basisin case of ties for exiting variables choose the one with smallest i
using one of these rules, the simplex algorithm terminates afterfinitely many steps
Definition 43 (Lexicographically Smaller)
A vector u ∈ Rn is said to be lexicographically larger (smaller) thananother vector v ∈ Rn if u 6= v and the first nonzero component of u− vis positive (negative).
VU Algorithmics M. Leitner 98
Introduction Geometry Simplex Duality
Finding an initial BFS
to start the simplex algorithm, an initial BFS is needed
sometimes straightforward: Ax ≤ b,b ≥ 0⇒ Ax + s = b⇒ (x, s) = (0,b)
in general, however, finding a BFS is not easy (→ auxiliary LP)
VU Algorithmics M. Leitner 99
Introduction Geometry Simplex Duality
Finding an initial BFSConsider the problem
min c′x
s.t. Ax = b
x ≥ 0
w.l.o.g. we assume b ≥ 0 (possibly multiply some constraints by −1)
introduce artificial variables y ∈ Rm
Auxiliary problem
minm∑i=1
yi
s.t. Ax + y = b
x, y ≥ 0
which has the BFS x = 0 and y = b.VU Algorithmics M. Leitner 100
Introduction Geometry Simplex Duality
The Two-Phase Simplex Method (Phase I)
1 possibly multiply some constraints by −1 so that b ≥ 0
2 add artificial variables y1, . . . , ym, if necessary and apply simplex toauxiliary problem with cost
∑mi=1 yi
3 if optimal cost in auxiliary problem is positive, original problem isinfeasible (termination)
4 if optimal cost in auxiliary problem is zero, feasible solution tooriginal problem has been found; if no artificial variable is in the finalbasis, eliminate artificial variables and columns to obtain feasiblebasis for original problem
5 Repeat: if lth basic variable is artificial, examine lth entry of thecolumns B−1Aj , j = 1, . . . , n; if all entries are zero, lth constraint isredundant and is eliminated; otherwise apply change of basis: lthbasic variable exists and xj enters (lth entry of jth column isnonzero)
VU Algorithmics M. Leitner 101
Introduction Geometry Simplex Duality
Two-Phase Simplex (Example from [1])
Initial LP
Auxiliary LP
VU Algorithmics M. Leitner 102
Introduction Geometry Simplex Duality
Two-Phase Simplex (Phase I, Example from [1])
BFS (x5, x6, x7, x8) = b = (3, 2, 5, 1), B = IcB = (1, 1, 1, 1), c i = −c′BAi
→ x4 enters the basis, x8 exits the basis
VU Algorithmics M. Leitner 103
Introduction Geometry Simplex Duality
Two-Phase Simplex (Phase I, Example from [1])
B = I, only 0th row of the tableau changes
→ x3 enters the basis, x4 exits the basis
VU Algorithmics M. Leitner 104
Introduction Geometry Simplex Duality
Two-Phase Simplex (Phase I, Example from [1])
→ x2 enters the basis, x6 exits the basis(degenerate pivot with θ∗ = 0)
VU Algorithmics M. Leitner 105
Introduction Geometry Simplex Duality
Two-Phase Simplex (Phase I, Example from [1])
→ x1 enters the basis, x5 exits the basis
VU Algorithmics M. Leitner 106
Introduction Geometry Simplex Duality
Two-Phase Simplex (Phase I, Example from [1])
cost is zero, i.e. feasible solution to the original problemx7 is still in the basis and we need to drive x7 out of the basisx7 is 3rd basis variable and 3rd entry of columns B−1Aj ,j = 1, . . . , 4, (original variables) is zeroindicates that A has linearly dependent rows and we can remove 3rdrow (redundant constraint)
VU Algorithmics M. Leitner 107
Introduction Geometry Simplex Duality
Two-Phase Simplex (Phase I, Example from [1])Initial tableau for original problem:
we may now compute reduced costs of the original variables andapply simplex to original problem
artificial variable x8 was unnecessary (since x4 appears in a singleconstraint only)
in this case (typical example: slack variable) we can let that variablein the initial basis
VU Algorithmics M. Leitner 108
Introduction Geometry Simplex Duality
Big-M Method
Auxiliary problem with y ∈ Rm
minn∑
j=1
cjxj + Mm∑i=1
yi
s.t. Ax + y = b
x ≥ 0
y ≥ 0
combines both faces into a single one
for sufficiently large M, artificial variables are driven to zero
VU Algorithmics M. Leitner 109
Introduction Geometry Simplex Duality
Computational Complexity of the Simplex Algorithm
computational effort at each iteration: O(mn)
number of iterations?
for all known variants, there are worst case examples such thatexponentially many extreme points are visited (Klee-Minty cube)in practice usually only O(m) pivots are needed
Important open question
Is it true that for every pivoting rule there are examples where thesimplex method takes an exponential number of iterations?
VU Algorithmics M. Leitner 110
Introduction Geometry Simplex Duality
Computational Complexity - Spanning Path
Figure from Bertsimas, Tsitsiklis, 1997
maxn∑
i=1
xi
ε ≤ x1 ≤ 1
εxi−1 ≤ xi ≤ 1− εxi−1 i = 2, . . . , n
→ unit cube has 2n vertices, simplex needs 2n − 1 pivots (costs arestrictly decreasing along path due to ε)
VU Algorithmics M. Leitner 111
Introduction Geometry Simplex Duality
Alternative Approaches
There are algorithms that solve LPs in polynomial time:
ellipsoid method: L. Khachian (1979): mainly of theoretical value(numerics, long running times)
interior point methods: N. Karmarkar (1984), partly competitive(no “warm start”), very important for theoretical research!
Simplex
warmstart is possible (e.g. after adding rows or columns)
has good practical (empirical) performance
is by far the most commonly used method in practice andincorporated in most commercial codes for LP
VU Algorithmics M. Leitner 112
Introduction Geometry Simplex Duality
Summary
simplex moves from on extreme point of the feasible set to another,each time reducing the cost, until an optimal solution is reached
cornerstones
optimality conditions (nonnegative of reduced costs) allow to testwhether current basis is optimalsystematic method for performing basis changes while not optimal
degeneracy causes substantial difficulties including the possibility ofcycling (since a change of basis may keep us at the same BFS, withno resulting cost improvement)
cycling can be avoided by choosing suitable pivoting rules
two-phase simplex can be used to obtain initial BFS
simplex method is a rather efficient algorithm in practice
VU Algorithmics M. Leitner 113
Introduction Geometry Simplex Duality
Duality Theory
VU Algorithmics M. Leitner 114
Introduction Geometry Simplex Duality
Motivation
Consider the Primal Problem
min c′x
s.t. Ax = b
x ≥ 0
with an optimal solution x∗ (assumed to exist).
Relaxed Problem:
g(p) = min c′x + p′(b− Ax)
x ≥ 0
where p is a price vector (with dimension of b) and p′(b−Ax) a penalty.
VU Algorithmics M. Leitner 115
Introduction Geometry Simplex Duality
Motivation
Since the relaxed problem allows for more options, we should haveg(p) ≤ c′x∗. Indeed,
g(p) = minx≥0
[c′x + p′(b− Ax)
]≤ c′x∗ + p′(b− Ax∗) = c′x∗
since x∗ is feasible and satisfies Ax∗ = b. Hence, each p yields a lowerbound g(p) for the optimal cost c′x∗.
Dual problem
“search for the tightest lower bound of the type” max g(p)
Main result in duality theory: Optimal cost in the dual problem is equalto the optimal cost c′x in the primal (Violating Ax = b has no value ifprices are chosen according to an optimal solution for the dual problem)
VU Algorithmics M. Leitner 116
Introduction Geometry Simplex Duality
Motivation
By definition of g(p) we have
g(p) = minx≥0
[c′x + p′(b− Ax)
]= p′b + min
x≥0(c′ − p′A)x
Note that
minx≥0
(c′ − p′A)x =
{0 if c′ − p′A ≥ 0′
−∞ otherwise
Thus, when maximizing g(p), we only need to consider values of p forwhich g(p) is not −∞.
VU Algorithmics M. Leitner 117
Introduction Geometry Simplex Duality
Motivation
Dual (Linear Programming) Problem
The dual problem is the same as the linear programming problem
max p′b
s.t. p′A ≤ c′
Note:
we started with equality constraints Ax = b and ended with noconstraints on the sign of the price vector p
VU Algorithmics M. Leitner 118
Introduction Geometry Simplex Duality
Motivation
We can replace inequality constraints (primal), i.e. Ax ≥ b, byAx− s = b, x ≥ 0, which can be written as
[A | −I][xs
]= b.
Leading to the dual constraints
p′ [A | −I] ≤[c′ | 0′
],
or equivalentlyp′A ≤ c′, p ≥ 0.
VU Algorithmics M. Leitner 119
Introduction Geometry Simplex Duality
Motivation
Also, if x is free rather than sign-constrained, we use
minx
(c′ − p′A)x =
{0 if c′ − p′A = 0′
−∞ otherwise
to end up with the constraint
p′A = c′
in the dual.
VU Algorithmics M. Leitner 120
Introduction Geometry Simplex Duality
Motivation
Construction of the dual of a minimization problem
we have a vector of parameters (dual variables) p
for every p we have a method for obtaining a lower bound on theoptimal primal cost
dual problem is a maximization problem that looks for the tightestlower bound
for some vectors p we get −∞ (no useful information)
→ we only need to maximize over those p leading to nontrivial lowerbounds
VU Algorithmics M. Leitner 121
Introduction Geometry Simplex Duality
The dual problem
Let A be a matrix with rows ai and columns Aj . Then the followingdefines a pair of primal and dual problems:
min c′x max p′b
s.t. aix ≥ bi i ∈ M1 s.t. pi ≥ 0 i ∈ M1
aix ≤ bi i ∈ M2 pi ≤ 0 i ∈ M2
aix = bi i ∈ M3 pi free i ∈ M3
xj ≥ 0 j ∈ N1 p′A′j ≤ cj j ∈ N1
xj ≤ 0 j ∈ N2 p′A′j ≥ cj j ∈ N2
xj free j ∈ N3 p′A′j = cj j ∈ N3
VU Algorithmics M. Leitner 122
Introduction Geometry Simplex Duality
Example
Primal Dual
min x1 + 2x2 + 3x3 max 5p1 + 6p2 + 4p3
s.t. − x1 + 3x2 = 5 s.t. p1 free
2x1 − x2 + 3x3 ≥ 6 p2 ≥ 0
x3 ≤ 4 p3 ≤ 0
x1 ≥ 0 −p1 + 2p2 ≤ 1
x2 ≤ 0 3p1 − p2 ≥ 2
x3 free 3p2 + p3 = 3
VU Algorithmics M. Leitner 123
Introduction Geometry Simplex Duality
Relations between primal and dual variables and constraints
PRIMAL minimize DUAL maximize
≥ bi ≥ 0constraints ≤ bi variables ≤ 0
= bi free
≥ 0 ≤ cjvariables ≤ 0 constraints ≥ cj
free = cj
Theorem 44
If we transform the dual into an equivalent minimization problem andthen form its dual, we obtain a problem equivalent to the originalproblem.
Or more compact: “The dual of the dual is the primal”.
VU Algorithmics M. Leitner 124
Introduction Geometry Simplex Duality
Duality Theorems
Theorem 45 (Weak duality)
If x is a feasible solution to the primal problem an p is a feasible solutionto the dual problem, then
p′b ≤ c′x.
Hence the cost of any dual solution is a lower bound for the optimal cost.
Corollary 46
(a) If the optimal cost in the primal is −∞ (unbounded), then the dualproblem must be infeasible.
(b) If the optimal cost in the dual is +∞ (unbounded), them the primalproblem must be infeasible.
VU Algorithmics M. Leitner 125
Introduction Geometry Simplex Duality
Duality Theorems
Corollary 47
Let x and p be feasible solutions to the primal and the dual, respectively,and suppose that p′b = c′x. Then x and p are optimal solutions to theprimal and the dual, respectively.
Theorem 48 (Strong duality)
If a linear programming problem has an optimal solution, so does itsdual, and the respective optimal costs are equal.
Theorem 49 (Complementary Slackness)
Let x and p be feasible solutions to the primal and the dual problem,respectively. They are optimal if and only if:
pi (a′ix− bi ) = 0 ∀i
(cj − p′Aj)xj = 0 ∀j
VU Algorithmics M. Leitner 126
Introduction Geometry Simplex Duality
Summary
for each LP, we can define a dual LP
the dual of the dual is the primal
strong duality theorem!
allows to develop the dual simplex algorithm (important for warmstart when solving LPs)
(more advanced) duality theory provides powerful tools for furtherinsights and understanding of LPs and polyhedra
VU Algorithmics M. Leitner 127