Menkes van den Briel Member of Research Staff NICTA and ANU menkes@nicta.com.au Combining Linear...

Menkes van den BrielMember of Research StaffNICTA and ANUmenkes@nicta.com.au

Combining Linear Programming Based Decomposition Techniques with Constraint Programming

CP-based column generation

Application Reference

Urban transit crew management

T.H. Yunes., A.V. Moura, C.C. de Souza. Solving very large crew scheduling problems to optimality. Proceedings of ACM symposium on Applied Computing, pages 446-451, 2000.

T.H. Yunes., A.V. Moura, C.C. de Souza. Hybrid column generation approaches for urban transit crew management problems. Transportation Science 39(2):273-288, 2005.

Travelling tournament

K. Easton, G.L. Nemhauser, and M.A. Trick. Solving the travelling tournament problem: A combined integer programming and constraint programming approach. Proceedings of Practice and Theory of Automated Timetabling, volume 2740 of Lecture Notes in Computer Science, pages 100-112. Springer, 2002.

Two-dimensional bin packing

D. Pisinger, M. Sigurd. Using decomposition techniques and constraint programming for solving the two-dimensional bin-packing problem. Journal on Computing 19(1):36-51, 2007.

Graph coloring S. Gualandi. Enhancing constraint programming-based column generation for integer programs. PhD thesis, Politechnico di Milano, 2008.

Constrained cutting stock

T. Fahle, M. Sellmann. Cost based filtering for the constrained knapsack problem. Annals of Operations Research 115(1):73-93, 2002.

Employee timetabling

S. Demassey, G. Pesant, L.M. Rousseau. A cost-regular based hybrid column generation approach. Constraints 11(4):315-333, 2006.

Wireless mesh networks

A. Capone, G. Carello, I. Filippini, S. Gualandi, F. Malucelli. Solving a resource allocation problem in wireless mess networks: A comparison between a CP-based and a classical column generation. Networks 55(3):221-233, 2010.

Multi-machine scheduling

R. Sadykov, L.A. Wolsey. Integer programming and constraint programming in solving a multimachine assignment scheduling problem with deadlines and release dates. Journal on Computing 18(2):209-217, 2006.

CP-based column generation

Application Reference

Airline crew assignment

U. Junker, S.E. Karisch, N. Kohl, B. Vaaben, T. Fahle, M. Sellmann. A framework for constraint programming based column generation. Proceedings of Principles and Practice of Constraint Programming, volume 1713 of Lecture Notes in Computer Science, pages 261-274, 1999.

T. Fahle, U. Junker, S.E. Karisch, N. Kohl, M. Sellmann, B. Vaaben. Constraint programming based column generation for crew assignment. Journal of Hueristics 8(1):59-81, 2002.

M. Sellmann, K. Zervoudakis, P. Stamatopoulos, T. Fahle. Crew assignment via constraint programming: integrating column generation and heuristic tree search. Annals of Operations Research 115(1):207-225, 2002.

Vehicle routing with time windows

L.M. Rousseau. Stabilization issues for constraint programming based column generation. Proceedings of Integration of AI and OR techniques in CP for Combinatorial Optimization, volume 3011 of Lecture notes in Computer Science, pages 402-408. Springer, 2004.

L.M. Rousseau, M. Gendreau, G. Pesant, F. Focacci. Solving VRPTWs with constraint programming based column generation. Annals of Operations Research 130(1):199-216, 2004.

CP-based Benders decomposition

Application Reference

Parallel machine scheduling

V. Jain, I.E. Grossmann. Algorithms for hybrid MILP/CP models for a class of optimization problems. INFORMS Journal on Computing 13(4):258-276, 2001.

Polypropylene batch scheduling

C. Timpe. Solving planning and scheduling problems with combined integer and constraint programming. OR Spectrum 24(4):431-448, 2002.

Call center scheduling

T. Benoist, E. Gaudin, B. Rottembourg. Constraint programming contribution to Benders decomposition: A case study. Principles and Practice of Constraint Programming, volume 2470 of Lecture Notes in Computer Science, pages 603-617. Springer, 2002.

Multi-machine scheduling

J.N. Hooker. A hybrid method for planning and scheduling. Principles and Practice of Constraint Programming, volume 3258 of Lecture Notes in Computer Science, pages 305-316. Springer, 2004.

J.N. Hooker. Planning and scheduling to minimize tardiness. Principles and Practice of Constraint Programming, volume 3709 of Lecture Notes in Computer Science, pages 314-327. Springer, 2005.

CP versus IP

CP IP

Variables Finite domain Continuous, Binary, Integer

Constraints Symbolic:alldifferentcumulative

Linear,algebraic:(+, –, *, =, ≤, ≥)

Inference Constraint propagation

LP relaxation

GlobalOptimal

LocalFeasible

CP versus IP

• “MILP is very efficient when the relaxation is tight and models have a structure that can be effectively exploited”

• “CP works better for highly constrained discrete optimization problems where expressiveness of MILP is a major limitation”

• “From the work that has been performed, it is not clear whether a general integration strategy will always perform better than either CP or an MILP approach by itself. This is especially true for the cases where one of these methods is a very good tool to solve the problem at hand. However, it is usually possible to enhance the performance of one approach by borrowing some ideas from the other”– Source: Jain and Grossmann, 2001

Outline

• Background• Introduction• Dantzig Wolfe decomposition• Benders decomposition• Conclusions

What is your background?

• Have implemented Benders and/or Dantzig Wolfe decomposition

• Have heard about Benders and/or Dantzig Wolfe decomposition

• Have seen Bender and/or Dances with Wolves

Things to take away

• A better understanding of how to combine linear programming based decomposition techniques with constraint programming

• A better understanding of column generation, Dantzig Wolfe decomposition and Benders decomposition

• A whole lot of Python code with example implementations

Helpful installations

• Python 2.6.x or 2.7.x – “Python is a programming language that lets you work more

quickly and integrate your systems more effectively”– http://www.python.org/getit/

• Gurobi (Python interface)– “The state-of-the-art solver for linear programming (LP), quadratic

and quadratically constrained programming (QP and QCP), and mixed-integer programming (MILP, MIQP, and MIQCP)”

– http://www.gurobi.com/products/gurobi-optimizer/try-for-yourself

• NetworkX– “NetworkX is a Python language software package for the

creation, manipulation, and study of the structure, dynamics, and functions of complex networks”

– http://networkx.lanl.gov/download.html

Abbreviations

• Artificial Intelligence (AI)• Constraint Programming (CP)• Constraint Satisfaction Problem (CSP)• Integer Programming (IP)• Linear Programming (LP)• Mixed Integer Programming (MIP)• Mixed Integer Linear Programming (MILP)• Mathematical Programming (MP)• Operations Research (OR)

Outline

• Background• Introduction• Dantzig Wolfe decomposition• Benders decomposition• Conclusions

What is decomposition?

• “Decomposition in computer science, also known as factoring, refers to the process by which a complex problem or system is broken down into parts that are easier to conceive, understand, program, and maintain”– Source: http://en.wikipedia.org/wiki/Decomposition_(computer_science)

• Decomposition in linear programming is a technique for solving linear programming problems where the constraints (or variables) of the problem can be divided into two groups, one group of “easy” constraints and another of “hard” constraints

“easy” versus “hard” constraints

• Referring to the constraints as “easy” and “hard” may be a bit misleading– The “hard” constraints need not be very difficult in themselves,

but they can complicate the linear program making the overall problem more difficult to solve

– When the “hard” constraints are removed from the problem, then more efficient techniques could be applied to solve the resulting linear program

Example

• Shortest path problem (P)

Min (i,j)A cijxij

s.t. 1 for i = s Source

j:(i,j)A xij – j:(j,i)A xji = 0 for iN – {s, t} Flow

-1 for i = t Sink

xij {0, 1}

• Resource constrained shortest path problem (NP-complete)

Min (i,j)A cijxij

s.t. 1 for i = s Source

j:(i,j)A xij – j:(j,i)A xji = 0 for iN – {s, t} Flow

-1 for i = t Sink

(i,j)A dijxij ≤ C Capacity

xij {0, 1}

G = (N, A), source s, sink t

Example

• Assignment problem (P)

Max i=1,…, m, j=1,…,n cijxij

s.t. j=1,…,n xij = 1 for 1 ≤ i ≤ m Job

i=1,…,m xij = 1 for 1 ≤ j ≤ n Machine

xij {0, 1}

• Generalized assignment problem (NP-complete)

Max i=1,…, m, j=1,…,n cijxij

s.t. j=1,…,n xij = 1 for 1 ≤ i ≤ m Job

i=1,…,m dijxij ≤ Cj for 1 ≤ j ≤ n Capacity

xij {0, 1}

m jobs, n machines

Example

• Consider developing a strategic corporate plan for several production facilities. Each facility has its own capacity and production constraints, but decisions are linked together at the corporate level by budgetary considerations

Common constraints

Facility 1

Facility 2

Facility n

Independentconstraints

“easy” versus “hard” variables

• Referring to the variables as “easy” and “hard” may be a bit misleading– The “hard” variables need not be very difficult in themselves, but

they can complicate the linear program making the overall problem more difficult to solve

– When the “hard” variables are removed from the problem, then more efficient techniques could be applied to solve the resulting linear program

Example

• Capacitated facility location problem (NP-complete)

Min i=1,…,n,j=1,…,m cijxij + j=1,…,m fjyj

s.t. i=1,…,m xij ≥ 1 for j = 1,…, n Demand

j=1,…,n dixij ≤ Ciyi for i = 1,…, m Roll

xij ≤ yi for i = 1,…, m j = 1,…, n Flow impl.

xij ≥ 0yi {0, 1}

m facilities, n customers

Example

• Consider solving a multi period scheduling problem. Each period has its own set of variables but is linked together through resource consumption variables

Independent variables

Com

mon

var

iabl

es Period 1

Period 2

Period n

Outline

• Background• Introduction• Dantzig Wolfe decomposition• Benders decomposition• Conclusions

• PrimalMin cxs.t. Ax ≥ b [y]

x ≥ 0

• DualMax yTbs.t. yTA ≤ c [x]

y ≥ 0

Background

• PrimalMin cxs.t. Ax ≥ b [y]

x ≥ 0

• DualMax bTys.t. ATy ≤ cT [x]

y ≥ 0

Background

cxc

A

x

Ax b

bTybT

AT cTATy

y

Travelling salesman

• G = (N, A), cost cij

0

1

2

3

4

5

6

7

8

9

x y

0 20 19

1 1 1

2 17 15

3 14 6

4 12 12

5 12 3

6 9 8

7 15 20

8 19 11

9 7 5

Travelling salesman

• G = (N, A), cost cij

x y

0 20 19

1 1 1

2 17 15

3 14 6

4 12 12

5 12 3

6 9 8

7 15 20

8 19 11

9 7 5

0

1

2

3

4

5

6

7

8

9

Cost 60.78

Travelling salesman

• Variablesxij is 1 if arc (i, j) is on the shortest tour, 0 otherwise

• FormulationMin (i,j)A cijxij

s.t. i:(i,j)A xij = 1 for j N Inflow

j:(i,j)A xij = 1 for i N Outflow

i,jS:(i,j)A xij ≤ |S| – 1 for S N Subtour

xij {0, 1}

Travelling salesman

• Variablesxij is 1 if arc (i, j) is on the shortest tour, 0 otherwise

• FormulationMin (i,j)A cijxij

s.t. i:(i,j)A xij = 1 for j N Inflow

j:(i,j)A xij = 1 for i N Outflow

xij {0, 1}

Example code

Travelling salesman

• G = (N, A), cost cij

x y

0 20 19

1 1 1

2 17 15

3 14 6

4 12 12

5 12 3

6 9 8

7 15 20

8 19 11

9 7 5

1

2

3

4

5

6

7

9

0

8

Subtour0, 2, 7

Travelling salesman

• G = (N, A), cost cij

x y

0 20 19

1 1 1

2 17 15

3 14 6

4 12 12

5 12 3

6 9 8

7 15 20

8 19 11

9 7 5

1

2

3

4

5

6

7

9

0

8

Subtour0, 8, 1, 9

Travelling salesman

• G = (N, A), cost cij

x y

0 20 19

1 1 1

2 17 15

3 14 6

4 12 12

5 12 3

6 9 8

7 15 20

8 19 11

9 7 5

1

2

3

4

5

6

7

9

0

8

Subtour0, 8, 2, 7

Travelling salesman

• G = (N, A), cost cij

x y

0 20 19

1 1 1

2 17 15

3 14 6

4 12 12

5 12 3

6 9 8

7 15 20

8 19 11

9 7 5

0

1

2

3

4

5

6

7

8

9

Cost 79.98

Travelling salesman

• G = (N, A), cost cij

x y

0 20 19

1 1 1

2 17 15

3 14 6

4 12 12

5 12 3

6 9 8

7 15 20

8 19 11

9 7 5

0

1

2

3

4

5

6

7

8

9

Cost 60.78

LPs with many constraints

• The number of constraints that are tight (or active) is at most equal to the number of variables, so even with many constraints (possibly exponential many) only a small subset will be tight in the optimal solution

Active

Non-active

A

A Ax b

Row generation in the primal…

cxc

x

y

bT

AT

… is column generation in the dual

bTy

cTATy

…and vice versa

x

c

A

cx

bAx

AT AT

ycT

bTybT

y

Column generation in the primal

Row generation in the dual

=

Resource constrained shortest path

• G = (N, A), source s, sink t, for each (i, j) A, cost cij, resource demand dij, and resource capacity C

1

2

3

4

5

6

1,10

10,3

1,7

2,2

1,2 10,1

1,1

12,3

2,3

5,7

i jcij, dij

Capacity = 14

Source: Desrosiers and Lübbecke, 2005

Resource constrained shortest path

• G = (N, A), source s, sink t, for each (i, j) A, cost cij, resource demand dij, and resource capacity C

1 6

1,10

10,3

1,7

2,2

1,2 10,1

1,1

12,3

2,3

5,7

i jcij, dij

Cost 13Demand 13

Capacity = 142

3

4

5

Resource constrained shortest path

• Variablesxij is 1 if arc (i, j) is on the shortest path, 0

otherwise

• FormulationMin (i,j)A cijxij

s.t. 1 for i = s Source

j:(i,j)A xij – j:(j,i)A xji = 0 for iN – {s, t} Flow

-1 for i = t Sink

(i,j)A dijxij ≤ C Capacity

xij {0, 1}

Example code

Resource constrained shortest path

• Variablesk is 1 if path k is the shortest path, 0 otherwise

• FormulationMin kK ckk

s.t. kK k = 1 Convex

kK dkk ≤ C Capacity

k ≥ 0

• Arc variables • Path variables

Arc versus path

1

2

3

4

5

6

1

2

3

4

5

6

1

2

3

4

5

6

1

2

3

4

5

6

Example code

Revised Simplex method

• Min cxs.t. Ax ≥ b

x ≥ 0

• Min z = cxs.t. Ax = b

x ≥ 0

• Let x be a basic feasible solution, such that x = (xB, xN) where xB is the vector of basic variables and xN is the vector of non-basic variables

Add slack variables

Revised Simplex method

• Min z = cxs.t. Ax = b

x ≥ 0

• Min z = cBxB + cNxN

s.t. BxB + ANxN = bxB, xN ≥ 0

• Min z = cBxB + cNxN

s.t. xB = B-1b – B-1ANxN xB, xN ≥ 0

Rearrange

x = (xB, xN), c = (cB, cN), A = (B, AN)

Revised Simplex method

• Min z = cBxB + cNxN

s.t. xB = B-1b – B-1ANxN xB, xN ≥ 0

• Min z = cBB-1b + (cN – cBB-1AN)xN

s.t. xB = B-1b – B-1ANxN xB, xN ≥ 0

Substitute

Revised Simplex method

• Min z = cBB-1b + (cN – cBB-1AN)xN

s.t. xB = B-1b – B-1ANxN xB, xN ≥ 0

• At the end of each iteration we have– Current value of non-basic variables xN = 0

– Current objective function value z = cBB-1b

– Current value of basic variables xB = B-1b

– Objective coefficients of basic variables 0– Objective coefficients of non-basic variables (cN – cBB-1AN) are the

so-called reduced costs– With a minimization objective we want non-basic variables with

negative reduced costs

Revised Simplex method

• Simplex algorithm1. Select new basic variable (xN to enter the basis)

2. Select new non-basic variable (xB to exit the basis)

3. Update data structures

Revised Simplex method

• Simplex algorithmxS = b (slack variables equal rhs)x\S = 0 (non-slack variables equal 0)while minj{(cj – cBB-1Aj)} < 01. Select new basic variable j : (cj – cBB-1Aj) < 0

2. Select new non-basic variable j’ by increasing xj as much as possible

3. Update data structures by swapping columns between matrix B and matrix AN

Example

• Min z = – x1 – 2x2

s.t. – 2x1 + x2 ≥ 2– x1 + 2x2 ≥ 7x1 ≥ 7x1, x2 ≥ 0

• Min z = – x1 – 2x2

s.t. – 2x1 + x2 + x3 = 2– x1 + 2x2 + x4 = 7x1 + x5 = 7x1 , x2, x3 , x4 , x5 ≥ 0

Add slack variables

• Simplex method • Revised Simplex method

Example

bsc

x1 x2 x3 x4 x5 rhs

-z - 1 -2 0 0 0 0

x3 -2 1 1 0 0 2

x4 -1 2 0 1 0 7

x5 1 0 0 0 1 3

bsc

x1 x2 x3 x4 x5 rhs

-z -5 0 2 0 0 4

x2 -2 1 1 0 0 2

x4 3 0 -2 1 0 3

x5 1 0 0 0 1 3

bsc

x3 x4 x5 rhs

-z 0 0 0 0

x3 1 0 0 2

x4 0 1 0 7

x5 0 0 1 3

bsc

x3 x4 x5 rhs

-z 2 0 0 4

x3 1 0 0 2

x4 -2 1 0 3

x5 0 0 1 3

x2

-2

1

2

0

x1

-5

-2

3

1

• Simplex method • Revised Simplex method

Example

bsc

x1 x2 x3 x4 x5 rhs

-z 0 0 -3/4

5/3 0 9

x2 0 1 -1/3

2/3 0 4

x1 1 0 -2/3

1/3 0 1

x5 0 0 2/3 -1/3

1 2bsc

x1 x2 x3 x4 x5 rhs

-z 0 0 0 1 2 13

x2 0 1 0 1/2 1/2 5

x1 1 0 0 0 1 3

x3 0 0 1 -1/2

3/2 3

bsc

x3 x4 x5 rhs

-z 2 0 0 9

x2 -1/3

2/3 0 4

x1 -2/3

1/3 0 1

x5 2/3 -1/3

1 2

bsc

x3 x4 x5 rhs

-z 0 0 0 13

x2 0 1/2 1/2 5

x1 0 0 1 3

x3 1 -1/2

3/2 3

x3

-3/4

-1/3

-2/3

2/3

Column generation

• Simplex algorithmxS = b (slack variables equal rhs)x\S = 0 (non-slack variables equal 0)while minj{(cj – cBB-1Aj)} < 01. Select new basic variable j : (cj – cBB-1Aj) < 0

2. Select new non-basic variable j’ by increasing xj as much as possible

3. Update data structures by swapping columns between matrix B and matrix AN

In column generation, rather than checking the reduced cost for each variable, a subproblem is

solved to find a variable with negative reduced cost

LPs with many variables

• The number of basic (non-zero) variables is at most equal to the number of constraints, so even with many variables (possibly exponential many) only a small subset will be in the optimal solution

A

xB xN

• (cN – cBB-1AN) < 0

• (cN – yTAN) < 0

Column generation

Substitute

• (cN – yTAN) < 0

• PrimalMin cxs.t. Ax ≥ b

x ≥ 0

• DualMax yTbs.t. yTA ≤ c

y ≥ 0

Column generation

x

c

A

cx

bAx

AT ATy cT

bTybT

y

Column with negative reduced cost

Row with violated rhs

Resource constrained shortest path

• Variablesk is 1 if path k is the shortest path, 0 otherwise

• FormulationMin kK ckk

s.t. kK k = 1 Convex

kK dkk ≤ C Capacity

k {0, 1}

• PrimalMin kK ckk

s.t. kK k = 1 []kK dkk ≤ C []k ≥ 0

• DualMax + C s.t. + dk ≤ ck [k]

= free ≤ 0

Resource constrained shortest path

Need to find a path for which ck – – dk < 0

Implicitly search all paths by optimizing Min (i,j)A (cij – dij)

s.t. Source, Flow, Sink

Resource constrained shortest path

• G = (N, A), source s, sink t, for each (i, j) A, cost cij, resource demand dij, and resource capacity C

1

2

3

4

5

6

1

10

1

2

1 10

1

12

2

5

i j(cij – dij)

Capacity = 14

Resource constrained shortest path

• MasterMin kK ckk

s.t. kK k = 1 Convex

kK dkk ≤ C Capacity

k ≥ 0

• SubproblemMin (i,j)A (cij – dij)xij

s.t. 1 for i = s Source

j:(i,j)A xij – j:(j,i)A xji = 0 for iN – {s, t} Flow

-1 for i = t Sink

• Add variable to master if (i,j)A (cij – dij)xij – < 0

Example code

• Roll width W, m orders of di rolls of length li, i = 1,…, m

Cutting stock

12

31

36

45

11 x

4 x

4 x

2 x

lidi

100

• Roll width W, m orders of di rolls of length li, i = 1,…, m

Cutting stock

12 12 31 45

12 12 36

12

12 12 31 45

31 31 36

36

12 12 12 12 36

12

31

36

45

11 x

4 x

4 x

2 x

lidi

Rolls 5

100

100

98

96

96

Cutting stock

• Variablesxik is the number of times order i is cut from roll kyk is 1 if roll k is used, 0 otherwise

• FormulationMin k=1,…,K yk

s.t. k=1,…,K xik ≥ di for i = 1,…, n Demand

i=1,…,n lixik ≤ Wyk for k = 1,…, K Roll

xik ≥ 0 and integeryk {0, 1}

Example code

Cutting stock

• Variablesk is the number of times cutting pattern k is used

• FormulationMin kK k

s.t. kK aikk ≥ di for i = 1,…, m Demand

k ≥ 0 and integer

Cutting stock

• Cutting pattern variables

12 12 36 36k

aik [2, 0, 2, 0]

12

31

36

45

11 x

4 x

4 x

2 x

12 12 31 45k

aik [2, 1, 0, 1]

• PrimalMin kK k

s.t. kK aikk ≥ di [i]k ≥ 0

• DualMax i=1,…,n dii s.t. i=1,…,n aiki ≤ 1 [k]

i ≥ 0

Cutting stock

Need to find a cutting pattern for which 1 – i=1,…,n aiki < 0

Implicitly search all cutting patterns by optimizing Max i=1,…,n aii

s.t. i=1,…,n liai ≤ Wai ≥ 0 and integer

• m items with value i and weight li, i = 1,…, m, maximum allowed weight W

Cutting stock

$0.50, 45lbs

$0.50, 36lbs

$0.33, 31lbs

$0.125, 12lbs 12

31

36

45

lii

0.125

0.33

0.50

0.50

100lbs 12 12 36 36

Cutting stock

• MasterMin kK k

s.t. kK aikk ≥ di for i = 1,…, m Demand

k ≥ 0

• SubproblemMax i=1,…,m aii

s.t. i=1,…,m liai ≤ Wai ≥ 0 and integer

• Add variable to master if 1 – aii < 0

Example code

Generalized assignment

• n jobs, m machines, cost cij, demand dij, capacity Ci

1

2

3

4

5

1

2

j i

36

34

Cj

cij, dij

Job 1 2

1 17, 8 23, 15

2 21, 15 16, 7

3 22, 14 21, 23

4 18, 23 16, 22

5 24, 8 17, 11

Generalized assignment

• n jobs, m machines, cost cij, demand dij, capacity Ci

Cost 95

30

29

1

2

3

4

5

1

2

j i

36

34

Cj

cij, dij

Job 1 2

1 17, 8 23, 15

2 21, 15 16, 7

3 22, 14 21, 23

4 18, 23 16, 22

5 24, 8 17, 11

Generalized assignment

• Variablesxij is 1 if job j is assigned to machine i, 0 otherwise

• FormulationMax i=1,…,m,j=1,…,n cijxij

s.t. i=1,…,m xij = 1 for 1 ≤ j ≤ n Job

j=1,…,n dijxij ≤ Ci for 1 ≤ i ≤ m Capacity

xij {0, 1}

Example code

Generalized assignment

• Variablesik is 1 if machine i has job assignment k, 0

otherwise

• FormulationMax i=1,…,m,k=1,…,Ki cikik

s.t. i=1,…,m,k=1,…,Ki aijkik = 1 for 1 ≤ j ≤ n Job

k=1,…,Ki ik = 1 for 1 ≤ i ≤ m Convexity

ik {0, 1}

Generalized assignment

• Job assignment variables

ik

aijk [1, 0, 1, 0, 1]

ik

aijk [0, 1, 0, 1, 0]

1

2

3

4

5

1

2

3

4

5

1

2

1

2

Generalized assignment

• FormulationMax i=1,…,m,k=1,…,Ki cikik

s.t. i=1,…,m,k=1,…,Ki aijkik = 1 for 1 ≤ j ≤ n Job

k=1,…,Ki ik = 1 for 1 ≤ i ≤ m Convexity ik {0, 1}

Common constraints

Machine 1

Machine 2

Machine n

Independentconstraints

• PrimalMax i=1,…,m,k=1,…,Ki cikik

s.t. i=1,…,m,k=1,…,Ki aijkik = 1k=1,…,Ki ik = 1 ik ≥ 0

• Dual Min j=1,…,n j + i=1,…,m i

s.t. j=1,…,n aijkj + i ≥ cik j = freei = free

Generalized assignment

Need to find a cutting pattern for which j=1,…,n (cik – aijkj ) – i > 0 for i = 1,…,m

Implicitly search all cutting patterns by optimizing Max j=1,…,n (cij – aijj )

s.t. j=1,…,n dijaij ≤ Ci

aij ≥ 0 and integer

$55.00, 8lbs

• n items with value j and weight dij, j = 1,…, n, maximum allowed weight W

Generalized assignment

$52.00, 23lbs

$51.00, 14lbs

$55.00, 15lbs

$44.00, 8lbs

36lbs

Job 1

1 44, 8

2 55, 15

3 51, 14

4 52, 23

5 55, 8

Job 2

1 40, 15

2 37, 7

3 43, 23

4 34, 22

5 41, 11

36

1

2

3

4

5

1

2

34

1

2

3

4

5

1

2

Generalized assignment

• MasterMax i=1,…,m,k=1,…,Ki cikik

s.t. i=1,…,m,k=1,…,Ki aijkik = 1 for 1 ≤ j ≤ n Job

k=1,…,Ki ik = 1 for 1 ≤ i ≤ m Convexity

ik {0, 1}

• Subproblem (for each machine i)Max j=1,…,n (cij – aijj ) s.t. j=1,…,n dijaij ≤ Ci

aij ≥ 0 and integer

• Add variable to master if j=1,…,n (cij – aijj ) – i > 0

Example code

History of column generation

1961: A linear programming approach to the cutting-stock problemP.C. Gilmore and R.E. Gomory

1963: A linear programming approach to the cutting-stock problem–Part IIP.C. Gilmore and R.E. Gomory

1960: Decomposition principle for linear programsG.B. Dantzig and P. Wolfe

“Credit is due to Ford and Fulkerson for their proposal for solving multicommoditynetwork problems as it served to inspire the present development.”

1958: A suggested computation for maximal multicommodity network flowsL.R. Ford and D.R. Fulkerson

1969: A column generation algorithm for a ship scheduling problemL.E. Appelgren

Solving integer programs by column generation

2000: On Dantzig-Wolfe decomposition in integer programming and ways to perform branching in a branch-and-price algorithmF. Vanderbeck

2005: A primer in column generationJ. Desrosiers and M.E. Lubbecke

1998: Branch-and-price: column generation for solving huge integer programsC. Barnhart, E.L. Johnson, G.L. Nemhauser, M.W.P. Savelsbergh and P.H. Vance

1984: Routing with time windows by column generationY. Dumas, F. Soumis and M. Desrochers

2011: Branching in branch-and-price: a generic schemeF. Vanderbeck

CP-based column generation

2000: Solving very large crew scheduling problems to optimalityT.H. Yunes, A.V. Moura and C.C. de Souza

1999: A framework for constraint programming based column generationU. Junker, S.E. Karisch, N. Kohl, B. Vaaben, T. Fahle and M. Sellmann

CP-based column generation

Application Reference CP used to solve subproblem

CP used within Branch-and-Price

Urban transit crew management

T.H. Yunes., A.V. Moura, C.C. de Souza. 2000.

Y Y

T.H. Yunes., A.V. Moura, C.C. de Souza. 2005.

Y Y

Travelling tournament

K. Easton, G.L. Nemhauser, and M.A. Trick. 2002.

Y Y

Two-dimensional bin packing

D. Pisinger, M. Sigurd. 2007. Y Y

Graph coloring S. Gualandi. 2008. Y Y

Constrained cutting stock

T. Fahle, M. Sellmann. 2002. Y N

Employee timetabling

S. Demassey, G. Pesant, L.M. Rousseau. 2006.

Y Y

Wireless mesh networks

A. Capone, G. Carello, I. Filippini, S. Gualandi, F. Malucelli. 2010.

Y N

Multi-machine scheduling

R. Sadykov, L.A. Wolsey. 2006. Y N

Source: Gualandi and Malucelli, 2009

CP-based column generation

Application Reference CP used to solve subproblem

CP used within Branch-and-Price

Airline crew assignment

U. Junker, S.E. Karisch, N. Kohl, B. Vaaben, T. Fahle, M. Sellmann. 1999.

Y N

T. Fahle, U. Junker, S.E. Karisch, N. Kohl, M. Sellmann, B. Vaaben. 2002.

Y N

M. Sellmann, K. Zervoudakis, P. Stamatopoulos, T. Fahle. 2002.

Y N

Vehicle routing with time windows

L.M. Rousseau. 2004. Y N

L.M. Rousseau, M. Gendreau, G. Pesant, F. Focacci. 2004.

Y Y

Source: Gualandi and Malucelli, 2009

CP-based column generation

• Typical implementation

Master Subproblem

Linearprogramming

Constraintprogramming

Dual information

New columns

Outline

• Background• Introduction• Dantzig Wolfe decomposition• Benders decomposition• Conclusions

Two-stage optimization

Stage 1 Stage 2

Solution values

Benders decomposition

Stage 1 Stage 2

Solutionvalues

Benderscuts

Benders decomposition

“Learn from ones mistakes” Distinguish primary variables from secondary variables Search over primary variables (master problem) For each trial value of primary variables, solve problem over

secondary variables (subproblem) If solution is suboptimal/infeasible, find out why and design a

constraint that rules out not only this solution but a large class of solutions that are suboptimal/infeasible for the same reason (Benders cut)

Add Benders cut to the master problem and resolve

Master Subproblem

Solutionvalues

Benderscuts

Capacitated facility location

• m facilities, n customers, cost cij, demand dj, capacity Ci, fixed cost fi

1

2

3

4

5

1

3

2

i jCi, fi

cijdj

10, 3

10, 4

10, 4

6

7

4

8

5

Cust 1 2 3

1 2 4 5

2 3 3 4

3 4 1 2

4 5 2 1

5 7 6 3

Capacitated facility location

• m facilities, n customers, cost cij, demand dj, capacity Ci, fixed cost fi

1

2

3

4

5

1

3

2

i jCi, fi

cijdj

10, 3

10, 4

10, 4

6

7

4

8

5

Cust 1 2 3

1 2 4 5

2 3 3 4

3 4 1 2

4 5 2 1

5 7 6 3

Cost 21.29

Capacitated facility location

• Variablesxij fraction of demand supplied by facility i to cusomter

jyi is 1 if facility i is open, 0 otherwise

• FormulationMin i=1,…,n,j=1,…,m cijxij + j=1,…,m fjyj

s.t. i=1,…,m xij ≥ 1 for j = 1,…, n Demand

j=1,…,n dixij ≤ Ciyi for i = 1,…, m Roll

xij ≤ yi for i = 1,…, m j = 1,…, n Flow

xij ≥ 0yi {0, 1}

Example code

Master SubproblemSolutionvalues

Benderscuts

• Min cx + dys.t. Ax ≥ b

Px + Qy ≥ rx ≥ 0 and integery ≥ 0

• Min cx + s.t. Ax ≥ b

x ≥ 0 and integer ≥ 0

•

Min dys.t. Qy ≥ r – Px

y ≥ 0

Benders decomposition

What if the subproblem is infeasible?

Benders decomposition

• Primal, dual possibilities

Optimal Unbounded

Infeasible

Optimal Yes No No

Unbounded

No No Yes

Infeasible No Yes Yes

Dual

Primal

Master SubproblemSolutionvalues

Benderscuts

• Min cx + dys.t. Ax ≥ b

Px + Qy ≥ rx ≥ 0 and integery ≥ 0

• Min cx + s.t. Ax ≥ b

optimality cutsfeasibility cutsx ≥ 0 and integer ≥ 0

•

Min dys.t. Qy ≥ r – Px

y ≥ 0

Benders decomposition

• Min dys.t. Qy ≥ r – Px [u]

y ≥ 0

• Optimal

• Infeasible

• Max uT(r – Px)s.t. uTQ ≤ d [y]

u ≥ 0

• Optimality cut ≥ uk

T(r – Px)

• Infeasibility cutvk

T(r – Px) ≤ 0

Benders decomposition

Master SubproblemSolutionvalues

Benderscuts

• Min cx + dys.t. Ax ≥ b

Px + Qy ≥ rx ≥ 0 and integery ≥ 0

• Min cx + s.t. Ax ≥ b

≥ ukT(r – Px)

vkT(r – Px) ≤ 0

x ≥ 0 and integer ≥ 0

•

Max dys.t. Qy ≤ r – Px

y ≥ 0

Benders decomposition

Benders decomposition

Solve master problem

Is optimal?

START

Solve sub problem

Terminate?

END

Add optimality cut

Add feasibility cut

yes no

yes

no

Capacitated facility location

• Variablesxij fraction of demand supplied by facility i to cusomter

jyi is 1 if facility i is open, 0 otherwise

• FormulationMin i=1,…,n,j=1,…,m cijxij + j=1,…,m fjyj

s.t. i=1,…,m xij ≥ 1 for j = 1,…, n Demand

j=1,…,n dixij ≤ Ciyi for i = 1,…, m Roll

xij ≤ yi for i = 1,…, m j = 1,…, n Flow

xij ≥ 0yi {0, 1}

Capacitated facility location

• Master Min j=1,…,m fjyj + s.t. optimality cuts

feasibility cutsyi {0, 1} ≥ 0

• SubproblemMin i=1,…,n,j=1,…,m cijxij s.t. i=1,…,m xij ≥ 1 for j = 1,…, n Demand

j=1,…,n dixij ≤ Ciyi for i = 1,…, m Roll

xij ≤ yi for i = 1,…, m j = 1,…, n Flow

xij ≥ 0

Capacitated facility location

• Subproblem primalMin i=1,…,n,j=1,…,m cijxij s.t. i=1,…,m xij ≥ 1 [j]

j=1,…,n dixij ≤ Ciyi [i]xij ≤ yi [ij]

xij ≥ 0

• Subproblem dualMax j=1,…,m j + i=1,…,n Ciyii + i=1,…,n,j=1,…,m yiij

s.t. j + dii + ij ≥ 1 [xij]j ≥ 0i ≤ 0ij ≤ 0

Capacitated facility location

• Master Min j=1,…,m fjyj + s.t. ≥ j=1,…,m j + i=1,…,n Ciiyi + i=1,…,n,j=1,…,m ij yi

j=1,…,m j + i=1,…,n Ciiyi + i=1,…,n,j=1,…,m ij yi ≤ 0yi {0, 1} ≥ 0

Example code

Benders decomposition for stochastic prog.

Master Scenario 2

Scenario 1

Scenario 3

Capacitated facility location

• m facilities, n customers, cost cij, demand dj, capacity Ci, fixed cost fi

1

2

3

4

5

1

3

2

i jCi, fi

cijdj

10, 3

10, 4

10, 4

6

7

4

8

5

Cust 1 2 3

1 2 4 5

2 3 3 4

3 4 1 2

4 5 2 1

5 7 6 3

5

6

3

7

4

4

5

2

6

3

Example code

CP-based Benders decomposition

• Typical implementation(?)

Master Subproblem

Constraintprogramming

Linearprogramming

Solution values

Benderscuts

CP-based Benders decomposition

• Recent developments

Master Subproblem

Integerprogramming

Constraintprogramming

Solution values

Benderscuts

CP-based Benders decomposition

Application Reference Master problem

Subproblem

Parallel machine scheduling

V. Jain, I.E. Grossmann. 2001. MILP CP

Polypropylene batch scheduling

C. Timpe. 2002. MILP CP

Call center scheduling

T. Benoist, E. Gaudin, B. Rottembourg. 2002.

CP LP

Multi-machine scheduling

J.N. Hooker. 2004. MILP CP

J.N. Hooker. 2005. MILP CP

Source: Hooker, 2006

Nested Benders decomposition

• Nested Benders decomposition– When the subproblem is decomposed into a master and

subproblem

Master Sub

Master Sub

Master Sub

Master Sub

Forward passSolve master

problems

Backward passSolve subproblems

and add Benders cuts

Outline

• Introduction• Background• Dantzig Wolfe decomposition• Benders decomposition• Conclusions

Why use decomposition?

• Many real-world systems contain loosely connected components, and as a result, the corresponding mathematical models present a certain structure that can be exploited

• It may be your only choice when solving the model without decomposition is impossible, because it is too large (memory error or timeout)

When is decomposition likely most effective?

• When you have either complicating constraints or complicating variables

Dantzig Wolfe decomposition

Bendersdecomposition

Further reading

• Column Generation– Guy Desaulniers, Jacques Desrosiers, Marius M. Solomon

• Decomposition Techniques in Mathematical Programming– Antonio J. Conejo, Enrique Castillo, Roberto Minguez and Raquel

Garcia-Bertrand

• Linear Programming and Network Flows– Mokhtar S. Bazaraa, John J. Jarvis, Hanif D. Sherali

From imagination to impact