R. Sadykov and F. Vanderbeck - CNR · R. Sadykov and F. Vanderbeck Column Generation for Extended Formulations Extended formulation in practice 1 Use a direct MIP-solver approach:

Column Generation for Extended Formulations

R. Sadykov and F. Vanderbeck

INRIA team ReAlOpt & University of Bordeaux

(inputs from L.A. Wolsey)

R. Sadykov and F. Vanderbeck Column Generation for Extended Formulations

Extended formulations: why & how

Reformulation involving extra variables

⇓

tighter relations between variables

Variable Splitting (binary or unary expansion)Network Flow (Multi-Commodity)Dynamic Programming Solver [Martin et al]Union of Polyhedra [Balas]Polyhedral Branching Systems [Kaibel, Loos]. . .

often rely on problem decomposition




⇓







⇓





Extended formulation in practice

1 Use a direct MIP-solver approach: size is an issue.

2 Use projection tools: Benders’ cuts.→ dynamic outer approximation of the intended form.

3 Use of an approximation [Van Vyve & Wolsey MP06]Drop some of the constraintsAggregate commoditiesPartial reformulation

→ static outer approximation of the intended form.

4 Use column generation (and row management)→ dynamic inner approximation of the intended form.























Outline

1 Assumption 1: ∃ Extended Formulation for a SPMachine SchedulingCapacitated Network Design

2 Assumption 2: ∃ Tight Reformulation for a SPBin Packing

3 Column-and-Row Generation

4 Recombination PropertyNetwork Flow based reformulationsDyn. Prog. based reformulationsUnion of Polyhedra based reformulations

5 Numerical experiments


Assumption 1: ∃ Extended Formulation for a SPOutline







Assumption 1: ∃ Extended Formulation for a SPAssumption 1: ∃ Extended Formulation for a SP

[F] ≡ min{c xA x ≥ aB x ≥ b

x ∈ INn}

Subproblem : P = {x ∈ Rn+ : Bx ≥ b} and X = P ∩ Zn

Extended Formulation for a Subproblem

∃ a polyhedron Q = {z ∈ Re+ : Hz ≥ h, z ∈ Re

+} and transformation T s.t.:Q defines an extended formulation for conv(X):

conv(X) = projx Q = {x = Tz : Hz ≥ h, z ∈ Re+}

Special Case: Dantzig-Wolfe reformulation, let X = {xg}g∈G

conv(X) = {x =X

g

xg λg :X

g

λg = 1, λg ≥ 0}




x ∈ INn}







conv(X) = {x =X

g

xg λg :X

g

λg = 1, λg ≥ 0}




x ∈ INn}







conv(X) = {x =X

g

xg λg :X

g

λg = 1, λg ≥ 0}


Assumption 1: ∃ Extended Formulation for a SPExtended or Dantzig-Wolfe reformulation


x ∈ INn}

[R] ≡ min{c T zA T z ≥ a

H z ≥ hz ∈ Ze

+}

[M] ≡ min{P

g∈Gc xg λgPg∈GA xg λg ≥ aP

g∈Gλg = 1

λ ∈ {0, 1}|G|}


Assumption 1: ∃ Extended Formulation for a SPRestricted reformulations


x ∈ INn}

[R] ≡ min{c T zA T z ≥ a

H z ≥ hz ∈ Ze

+}

[M] ≡ min{P


g∈Gλg = 1

λ ∈ {0, 1}|G|}


Assumption 1: ∃ Extended Formulation for a SPHybrid Approach: two points of view

1 An alternative to a direct extended formulation approach

Dynamic generation of the variables of [R]:generated in bunch by optimizing over the SP

Adding rows that become active.

2 An alternative to standard column generation

Perform Column Generation for [M]

“Project” the Master Program in [R]


Assumption 1: ∃ Extended Formulation for a SPSingle Machine Scheduling: reformulations

[F] ≡ min{X

j

c(Sj ) : Sj + pj ≤ Si or Si + pi ≤ Sj ∀(i, j) ∈ J × J}

timeB4 C3 D5C3A8

Sa Sb Sc Sd

[R] ≡ min{X

jt

cjt zjt

T−pj +1Xt=1

zjt = 1 ∀j

Xj

tXτ=t−pj +1

zjτ ≤ 1 ∀t

zjt ∈ {0, 1} ∀j, t}

[M] ≡ min{Xg∈G

cgλg :Xg∈G

T−pj +1Xt=1

zgjtλg = 1 ∀j,

Xg∈G

λg = 1, λg ∈ {0, 1} ∀g ∈ G}



[F] ≡ min{X

j


timeB4 C3 D5C3A8

Sa Sb Sc Sd

[R] ≡ min{X

jt

cjt zjt

T−pj +1Xt=1

zjt = 1 ∀j

Xj

tXτ=t−pj +1

zjτ ≤ 1 ∀t

zjt ∈ {0, 1} ∀j, t}

[M] ≡ min{Xg∈G

cgλg :Xg∈G

T−pj +1Xt=1

zgjtλg = 1 ∀j,

Xg∈G

λg = 1, λg ∈ {0, 1} ∀g ∈ G}



[F] ≡ min{X

j


timeB4 C3 D5C3A8

Sa Sb Sc Sd

[R] ≡ min{X

jt

cjt zjt

T−pj +1Xt=1

zjt = 1 ∀j

Xj

tXτ=t−pj +1

zjτ ≤ 1 ∀t

zjt ∈ {0, 1} ∀j, t}

[M] ≡ min{Xg∈G

cgλg :Xg∈G

T−pj +1Xt=1

zgjtλg = 1 ∀j,

Xg∈G

λg = 1, λg ∈ {0, 1} ∀g ∈ G}


Assumption 1: ∃ Extended Formulation for a SPSingle Machine Scheduling: Hybrid Approach

1 Generate a column for [M]:

[SP] ≡ min{X

jt

(cjt−πj )zjt :X

j

tXτ=t−pj +1

zjτ ≤ 1 ∀t, zjt ∈ {0, 1} ∀j, t},

1 2 3 4 5 6 7 8

2 Disaggregate the SP solution in arc variables z for [R]

3 Add the associated flow conservation constraints to [R]

4 Solve the restricted [R] to obtain dual prices




[SP] ≡ min{X

jt

(cjt−πj )zjt :X

j

tXτ=t−pj +1

zjτ ≤ 1 ∀t, zjt ∈ {0, 1} ∀j, t},

1 2 3 4 5 6 7 8







[SP] ≡ min{X

jt

(cjt−πj )zjt :X

j

tXτ=t−pj +1

zjτ ≤ 1 ∀t, zjt ∈ {0, 1} ∀j, t},

1 2 3 4 5 6 7 8







[SP] ≡ min{X

jt

(cjt−πj )zjt :X

j

tXτ=t−pj +1

zjτ ≤ 1 ∀t, zjt ∈ {0, 1} ∀j, t},

1 2 3 4 5 6 7 8





Assumption 1: ∃ Extended Formulation for a SPStandard Column Gen. versus Hybrid Approach

Iteration Subproblem solution

Initial solution

· · · · · ·

Final solution

Column generation for [M]

1

2

3

10

11

Column-and-rowgeneration for [R]

Subproblem solution


Assumption 1: ∃ Extended Formulation for a SPInterest of the Hybrid Approach: Flow recombination

T = 7 pj = j

for g = 1, xgjt = 1 for (j, t) ∈ {(3, 1), (2, 4), (2, 6)};

for g = 2, xgjt = 1 for (j, t) ∈ {(5, 1), (1, 6), (1, 7)}.

In the associated [R] formulation, these solutions can be recombined:

1 2 3 4 5 6 7 8

But such recombinations are not feasible in the restricted master [M].


Assumption 1: ∃ Extended Formulation for a SPInterest of the Hybrid Approach: Numerical Tests

pj ∈ U[1, 10]

n Iter[R] Time[R] Iter[M] Time[M]12 29 0.4 92 1.625 50 3.7 351 36.250 87 33.3 1336 762.6

Iteration decrease: factor 15; time decrease: factor 22.

pj ∈ U[1, 50]

n Iter[R] Time[R] Iter[M] Time[M]12 116 16.6 121 46.025 174 150.6 399 876.4

Iteration decrease: factor 2.3; time decrease: factor 6.


Assumption 1: ∃ Extended Formulation for a SPMulti-Commodity Capacitated Network Design

[F] ≡ min{Xijk

ckij xk

ij +X

ij

fij yijXj

xkji −

Xj

xkij = dk

i ∀i, k

Xk

xkij ≤ uij yij ∀i, j

xkij ≥ 0 ∀i, j, k

yij ∈ IN ∀i, j}

[SP ij ] ≡ min{X

k

ck xk + f y :Xk

xk ≤ u y

xk ≤ min{dk , u y}∀k}


Assumption 1: ∃ Extended Formulation for a SPNetwork Design: Extended form. for the SPs

Let ysij = 1 and xks

ij = xkij if yij = s.

[SP ij ] ≡ min{Xks

ckij xks

ij +X

s

fij s ysij :X

s

ysij ≤ 1

(s − 1) uij ysij ≤

Xk

xksij ≤ s uij ys

ij ∀s

xksij ≤ min{dk , s uij} ys

ij ∀k , s}

Extended f. for SP [Croxton, Gendron and Magnanti OR07]


Assumption 1: ∃ Extended Formulation for a SPNetwork Design: extended formulation

[R] ≡ min{Xijks

ckij xks

ij +Xijs

fij s ysijX

js

xksji −

Xjs

xksij = dk

i ∀i, k

(s − 1) uij ysij ≤

Xk

xksij ≤ s uij ys

ij ∀i, j, s

0 ≤ xksij ≤ dk ys

ij ∀i, j, k , sXs

ysij = 1 ∀i, j

ysij ∈ {0, 1} ∀i, j, s}

solved by col-and-row generation [Frangioni & Gendron DAM09]better performance than by adding Benders’ cut to [F]


Assumption 1: ∃ Extended Formulation for a SPNetwork Design: standard col. gen. formulation

[M] ≡ min{X

i,j,s,g∈Gij

(ckij xg

ks + fij s ygs ) λij

g

Xjs

Xg∈Gij

xgks λ

ijg −

Xjs

Xg∈Gij

xgks λ

ijg = dk

i ∀i, k

Xg∈Gij

λijg ≤ 1 ∀i, j

λijg ∈ {0, 1} ∀i, j, g ∈ Gij}

col-and-row generation for [R] outperforms standard col gen for [M][Frangioni & Gendron WP10]


Assumption 1: ∃ Extended Formulation for a SPNetwork Design: Union of Polyhedra with [R]






Assumption 1: ∃ Extended Formulation for a SPNetwork Design: Union of Polyhedra with [M]


Assumption 2: ∃ Tight Reformulation for a SPOutline







Assumption 2: ∃ Tight Reformulation for a SPAssumption 2: ∃ Tight Reformulation for a SP


x ∈ INn}


Reformulation for a Subproblem


+} and transformation T s.t.:Q defines an tighter formulation for X :

conv(X)⊂projx Q = {x = Tz : Hz ≥ h, z ∈ Re+}⊂P


Assumption 2: ∃ Tight Reformulation for a SPBin Packing

[F] ≡ min{X

k

δk :Xk

xik = 1 ∀iXi

si xik ≤ C δk ∀k

xik ∈ {0, 1} ∀i, kδk ∈ {0, 1} ∀k}


Assumption 2: ∃ Tight Reformulation for a SPBin Packing: standard column generation

[F] ≡ min{X

k

δk :Xk

xik = 1 ∀iXi

si xik ≤ C δk ∀k

xik ∈ {0, 1} ∀i, kδk ∈ {0, 1} ∀k}

[SP] ≡ min{δ −X

i

πixi :X

i

sixi ≤ C δ, (x, δ) ∈ {0, 1}n+1}

[M] ≡ min{X

g

λg :

Xg

xgi λg = 1 ∀i ∈ I

λg ∈ {0, 1} ∀g ∈ G.}


Assumption 2: ∃ Tight Reformulation for a SPBin Packing: Network Flow Reformulation

[SP] ≡ min{δ −Xiuv

πi f iuv : (f , δ) ∈ {0, 1}n∗m+1

Xi,v

f i0v + f0C = δ

Xi,u

f iuv =

Xi,u

f ivu + fv,C v = 1, . . . ,C − 1

Xi,u

f iuC +

Xv

fvC = δ

0 ≤ f iuv ≤ 1 ∀i, u, v = u + si }

60 1 2 3 4 5

A relaxation to an unbounded knapsack Problem


Assumption 2: ∃ Tight Reformulation for a SPBin Packing: Network Flow ReformulationLet F i

uv =P

k f ikuv , FvC =

Pk f k

vC , and ∆ =P

k δk .

60 1 2 3 4 5

[R] ≡ min{∆ :X(u,v)

F iuv = 1 ∀i

Xi,v

F i0v + F0C = ∆

Xi,u

F iuv =

Xi,u

F ivu + FvC v = 1, . . . ,C − 1

Xi,u

F iuC +

Xv

FvC = ∆

F iuv ∈ {0, 1} ∀i, (u, v) : v = u + si }.

[Valerio de Carvalho AOR99]R. Sadykov and F. Vanderbeck Column Generation for Extended Formulations

Assumption 2: ∃ Tight Reformulation for a SPPros and Cons of the Hybrid Approach

“+”Recombinations of SP solutionsExtra variables for branching [Valerio de Carvalho AOR99]Extra variables for defining cuts [Uchoa et al]

“-”need to handle dynamic row generationrequires a specific pricing oracle in the extended spacesymmetries

60 1 2 3 4 5

60 1 2 3 4 5


Column-and-Row GenerationOutline







Column-and-Row GenerationRestricted reformulations

X = {xg}g∈G = {T zg}g∈G

[R] ≡ min{c T zA T z ≥ a π

H z ≥ hz ∈ Ze

+}

[M] ≡ min{P


g∈Gλg = 1

λ ∈ {0, 1}|G|}


Column-and-Row GenerationHybrid column generation: convergence

vRLP

iteration

restricted master Lp values

Master LP value


Column-and-Row GenerationHybrid column generation: convergence

L(π) = π a + minx∈X{(c − πA) x}

iteration

restricted master Lp values

intermediate Lagrangian bounds

Master LP value


Column-and-Row GenerationHybrid column generation procedure

Step 0: Initialize the dual bound, β := −∞, and the subproblemsolution set G so that the linear relaxation of [R] is feasible.

Step 1: Solve [RLP ] and collect the dual solution π.

Step 2: Solve the pricing problem: z∗ ←min{(c− πA) T z : z ∈ Z} = min{(c− πA) x : x ∈ X}.

Step 3: Compute the Lagrangian dual bound:L(π) = π a + (c − πA) T z∗, and update the dual boundβ := max{β, L(π)} (lagrangian dual value estimator).If vR

LP ≤ β, STOP.

Step 4: Update the current bundle G by adding solution zs := z∗ andupdate the resulting restricted reformulation [R]. Goto Step 1.

Proposition

Either vRLP ≤ β (stopping condition), or [(c − πA) T − σ H]z∗ < 0 and

some of the components of z∗ have negative reduced cost in [RLP ].


Column-and-Row GenerationHybrid column generation procedure

Step 0: Initialize the dual bound, β := −∞, and the subproblemsolution set G so that the linear relaxation of [R] is feasible.

Step 1: Solve [RLP ] and collect the dual solution π.

Step 2: Solve the pricing problem: z∗ ←min{(c− πA) T z : z ∈ Z} = min{(c− πA) x : x ∈ X}.

Step 3: Compute the Lagrangian dual bound:L(π) = π a + (c − πA) T z∗, and update the dual boundβ := max{β, L(π)} (lagrangian dual value estimator).If vR

LP ≤ β, STOP.

Step 4: Update the current bundle G by adding solution zs := z∗ andupdate the resulting restricted reformulation [R]. Goto Step 1.

Proposition

Either vRLP ≤ β (stopping condition), or [(c − πA) T − σ H]z∗ < 0 and

some of the components of z∗ have negative reduced cost in [RLP ].


Recombination PropertyOutline







Recombination PropertyWhy consider Col Gen for [R] instead of [M]

RLP ⊃ MLP

Property (recombination)

Given G ⊂ G, ∃z ∈ RLP(G), such that z 6∈ conv(Z (G)).

1 2 3 4 5 6 7 8

True for Network Flow based reformulations: w = z1 − z2

is a cycle flow; w decomposes into elementary cycle flow wA,wB, . . .; z = z1 + αwA ∈ Q; but, z 6∈ conv(z1, z2)


Recombination PropertyWhy consider Col Gen for [R] instead of [M]

RLP ⊃ MLP

Property (recombination)

Given G ⊂ G, ∃z ∈ RLP(G), such that z 6∈ conv(Z (G)).

1 2 3 4 5 6 7 8

True for Network Flow based reformulations: w = z1 − z2

is a cycle flow; w decomposes into elementary cycle flow wA,wB, . . .; z = z1 + αwA ∈ Q; but, z 6∈ conv(z1, z2)


Recombination PropertyTrue for DP based reformulations[Martin et al OR90]Consider a solution to dynamic programming recursion

γ(l) = min(J,l)∈A

{Xj∈J

γ(j) + c(J, l)}

21

7

1

2

3

4

5

6

8

9

10

11

12

13

14

15

16

17

18

19

20


Recombination PropertyTrue for DP based reformulations

The recombination of DP sol 1 and DP sol 2 into DP sol 3

21

7

1

2

3

4

5

6

8

9

10

11

12

13

14

15

16

17

18

19

20

21

7

1

2

3

4

5

6

8

9

10

11

12

13

14

15

16

17

18

19

20

21

7

1

2

3

4

5

6

8

9

10

11

12

13

14

15

16

17

18

19

20


Recombination PropertyFor Union of Polyhedra based reformulations

Qk = {zk ∈ Re+ : Hkzk ≥ hk ; zk ≤ uk}

Q = {z =∑

k

zk ;∑

k

δk = 1; zk ≤ uk δk∀k ,Hkzk ≥ hk δk∀k}


Numerical experimentsOutline







Numerical experimentsMulti Item Lot Sizing – small bucket

Periods 1 to 5

Machine

Item 2

Item 1

[F] ≡ min{X

kt

(ckt xk

t + f kt yk

t ) :Xk

ykt ≤ 1 ∀t

tXτ=1

xkτ ≥ Dk

1T ∀k , t

xkt ≤ Dk

tT ykt ∀k , t

xkt ≥ 0 ∀k , t

ykt ∈ {0, 1} ∀k , t}


Numerical experimentsMulti Item Lot Sizing – small bucket

[R] ≡ min{Xktu

cktu zk

tu :

Xk

TXu=t

zktu ≤ 1 ∀t,

Xu≥t

z0kt = 1 ∀k,

z0kt +

Xu<t

zku,t−1 =

Xu≥t

zktu ∀k, t = 1, . . . T ,

TXt=1

zkt,T = 1 ∀k,

zktu ∈ {0, 1} ∀k, u, t}

[M] ≡ min{X

k,g∈Gk(ck xg + f k yq ) λk

g :X

k,g∈Gkyg

t λkg ≤ 1 ∀t,

Xg∈Gk

λkg = 1 ∀k, λ ∈ IN|G|×K}

(K , T ) It[R] Sp[R] T[R] It[M] Sp[M] T[M](5,40) 65 109 2 109 540 9

(10,80) 116 341 25 166 1643 151

Iteration decrease: factor 1.4; time decrease: factor 6.


Numerical experimentsConclusion

1 Column generation for an extended formulation isto be considered when:

The extended formulation← decomposition principle,SP solutions can be recombined into alternative ones.

2 There are computational evidence in the literature thatthis can be a competitive approach. There are also studieswhere it could have been used (and wasn’t).

3 The approach can be interpreted a stabilization methodfor column generation:

disaggregation helps,related to the use of exchange vectors.

4 It can be applied to an approximation of an extendedformulation [Van Vyve & Wolsey MP06] (see bin packing)


Documents

R. Sadykov and F. Vanderbeck - CNR · R. Sadykov and F. Vanderbeck Column Generation for Extended Formulations Extended formulation in practice 1 Use a direct MIP-solver approach: