1

“Applications” of knapsack problems

10 September 2002

2

“Applications” of knapsack problems

1. What the hell is a knapsack?

2. Knapsack cover inequalities; lifting; surrogate duality and other boring definitions

3. Knapsacks and infeasibility

4. Knapsacks and (bounded) general integer variables

3

A knapsack problem is simply an integer programme with a single constraint

Max cx

Subject to:

ax ≤b

xBn

• Suppose you have a knapsack, with size b, and a bunch of items with size a i, and value ci, and wish to put the most value into the knapsack…

• … and can somehow mash the items so that they fit whenever the sum of their volumes fit…

4

Pure binary knapsack:

Max cx

Subject to:

ax ≤b

x Bn

Bounded knapsack:

Max cx

Subject to:

ax ≤b

0≤x≤u

x Zn

… they come in several flavours

Unbounded knapsack:

Max cx

Subject to:

ax ≤b

x≥0

x Zn

Bounded mixed integer knapsack:

Max cx+dy

Subject to:

ax+zy ≤b

x≥0

x Zn

y Rm

Bounded knapsacks are our focus here, though most results

are or appear to be readily generalisable to bounded mixed

integer knapsacks

5

… and can be used to solve all integer programming problems (if you know the right multiplier)…

Surrogate Duality

Maxx cx

Subject to:

Ax ≤b

x Bn

≡

Minu

(Maxx cx

Subject to:

uAx≤ub

x Bn )

u Rn+

• Any integer programme with many constraints…

• … can be solved by finding the right knapsack problem.

• uAx≤ub is called a surrogate knapsack

• …trouble is, finding the u that minimises this problem is difficult, and (a common drawback with dual approaches) the solution to the inner problem is not feasible at any stage until we find an optimal u

6

… in psuedo-polynomial time!

Solving a knapsack problem in pseudo-polynomial time:

Dynamic Progamming

Let KP(n,b)=

Maxx c1x1+c2x2+…+cnxn

s.t.

a1x1+a2x2+…+anxn≤b

Now:

KP(0,b’)=0 (Base Case 1)

KP(k,l), l<0 = -∞ (Base Case 2)

KP(n,b) = max(KP(n-1,b), cn+KP(n-1,b-an)) (Recursive case)

So, assuming the KP is scaled such that the a i are integral, the worst case solution time is proportional to nb.

7

… or can at least help solve them… (when you can’t find the right multiplier)

Surrogate duality: insane method

• Find the right multiplier u.

• Generate all the facets (x≤) of:

uAx≤ub

x Bn

• Solve the linear programme:

max cx

Subject to:

x≤

x Bn

• Finding all the facets (almost never pseudo- polynomial!) seems insane given we can just solve uAx<=ub by dynamic programming

• … HOWEVER ….

• Its hard to find the optimal multiplier u* - and because it’s a dual method, if we don’t find u* we don’t even have a feasible solution to the primal problem.

• Often its easy to find a “probably fairly close to optimal” multiplier u’ (We’ll see some approaches shortly)

• If u’ is close to u* then we expect:

– Some of the facets of u’ and u* may coincide – even getting some of the facets may cut off fractional values in LP based branch-and-bound

– Facets of u’ to coincide with facets of u* in many dimensions

8

What’s a valid inequality/facet/dimension?

9

… inside a Branch-and-bound-and-cut-and-preprocess-and-probe-and-run-a-primal-heuristic-and-variable-fix-and-whatever algorithm

Surrogate cutting planes:

Inside a branch-and-bound code:

Find a half decent multiplier u’.

• Generate a facet (x≤ of:

u’Ax≤u’b

x Bn

• Solve the linear programme:

max cx

Subject to:

Ax≤b

x≤

x Bn

• Repeat so long as the x≤’s are doing a reasonable job of cutting of fractional LP’s.

• The remainder of our discussion will focus on:

– How to “find a half decent multiplier”

– How to “Generate a facet”

10

The facets (or high dimension inequalities) we are interested in are called (lifted) knapsack cover inequalities.

jj C

a b

jj S C

a b

| | 1ij C

x C

{ | 0, }Z jP x j C x

{ | }nZP Z b x ax

For binary spaces:• A cover C is a set of items that don’t fit into the knapsack.

• A minimal cover C is a cover that “only just” doesn’t fit – i.e. every strict subset of those items will fit, but the set C won’t.

• A Knapsack cover inequality (KCI) simply states that therefore one of the items must not be in the knapsack.

• If C is minimal then the KCI is facet-defining for the space

• Lifting can then be use to come up with coefficients for the other variables so that we get a facet of the full space

11

Generation of Knapsack Cover inequalities is frequently applied to solve sparse binary problems…

• An integer programme where each variable occurs in only one constraint is separable into the individual knapsack constraints.

A=[ ]• A sparse constraint matrix is typically “close to separable” e.g.

A=[ ]Essentially this suggests that u’=(1,0,0,0,0…), u’=(0,1,0,0,0,…) etc. are pretty reasonable choices for u’.

Just solve each of the separate knapsack problems to solve the overall integer programme

Here we can’t just solve the separate knapsack problems, but a facet for any of the knapsacks is valid for the whole problem, and because there is little overlap, is likely to be either facet defining, or of high dimension for the whole problem.

12

… and has been very successful at reducing the computational effort required

# nodes

1

10

100

1000

10000

100000

1000000

With KCI w ithout

CPU time (s)

0.1

1

10

100

1000

10000

100000

With KCI w ithout

13

Knapsacks and infeasibility

• Most cutting plane algorithms generate cuts aiming to cut off the current LP solution at a feasible node of the branch and bound tree…

• … but that’s boring- we can also generate useful cutting planes at infeasible nodes

• … or nodes being pruned

• … and get a reasonable performance boost

14

Most cutting plane algorithms generate cuts aiming to cut off the current LP solution at a feasible node of the branch and bound tree…

Cut separation- e.g. Binary knapsack cover inequalities

Given the current LP solution xlp

Find the minimal Cover C which maximises:

(i.e. maximise the amount that the KCI is violated by the current LP)

Then lift it.

| |ii C

xlp C

15

… but that’s boring- we can also generate useful cutting planes at infeasible nodes…

• Recall an infeasible primal unbounded dual a divergent ray (r,z) exists for the current LP:

Min cx

s.t.

Ax=b

xi=1, iS1 (these constraints added by branching. W.lo.g. we assume all the branches were 1-branches)

[r is the multiplier for the original constraints, z for the branching constraints]

And with some trickery based on the properties:

i) rb-z.1 < 0 (improves objective)

ii) rA-zIB 0 (definition of a ray)

of a divergent ray, we find that:

Bz = B {i | zi0}

is a cover for the surrogate knapsack rAx ≤ rb :

rAx zIBx, for any x 0 (ii)

z.1 for any x | xi=1, i Bz (i)

> rb

16

… or nodes being pruned

• Let zBEST be the best integer solution we’ve found so far.

• Temporarily add the constraint:

cx≤zBEST

to the formulation at nodes being pruned.

• … now they are infeasible and we can apply the previous method.

• [Actually implementation-wise its quite different, as we’d rather not keep pivotting until the LP solver proved it was infeasible with the new constraint]

So, at an infeasible node, there’s no LP we’re trying to cut off-how can this approach help?!?

Example

After adding four branching constraints:

x1=1, x2=1, x3=1, x4=1

An infeasible node is found.

The divergent dual ray (r,z) has z=(0,0,23,12), so Bz={3,4}

And the constraint x3+x4≤1 is added to the formulation.

…This does nothing useful at the node that causes it to be added.

… but it does stop us exploring nodes with branching constraints:

x1=0, x2=1, x3=1, x4=1

x1=1, x2=0, x3=1, x4=1

x1=0, x2=0, x3=1, x4=1

17

… and get a moderate performance boost

# nodes

0

5000

10000

15000

20000

25000

30000

35000

40000

45000

IB defaults

CPU time (s)

0

2000

4000

6000

8000

10000

12000

14000

16000

With KCI w ithout

18

Knapsacks and (bounded) General Integer Variables

• Whilst you can always convert general integer variables to binary… many of us are not that patient.

• Ceria et. al. made a start generalising KCI to general integers

• Extending this approach with some much less natural generalisations…

– Ceria constraints aren’t necessarily facet defining for the cover space…

– … but lifted SMD subcovers are

– On general integers “minimal” is sometimes too small…

– … so we redefine minimal to be less small

• … and get significant computational improvements

19

Whilst you can always convert general integer variables to binary… many of us are not that patient.

log(CPU time)

0.01

0.1

1

10

100

1000

10000

Binary GI

log(#nodes)

1

10

100

1000

10000

100000

1000000

Binary GI

20

Ceria et. al. made a start generalising KCI to general integers….

• A cover C is a set of items that don’t fit into the knapsack.

• A minimal cover C is a cover that “only just” doesn’t fit – i.e. every strict subset of those items will fit, but the set C won’t.

• A Knapsack cover inequality (KCI) simply states that therefore some of one of the items must not be in the knapsack.

• If C is minimal then the KCI is facet-defining for the space P=blah

• Lifting can then be use to come up with coefficients for the other variables so that we get a facet of the full space P=blah

jj C

a b

jj S C

a b

| | 1ij C

x C

j jj C

a u b

j jj C

a u b

1ji i a

j C j C

x u

BINARY:

BOUNDED INTEGER (Ceria et. al.):

21

… but they aren’t facet defining for the cover space…

Example

Consider:

3x1+5x2+7x3 ≤ 50, 0 ≤ xi ≤4

C={1,2,3} is the only cover (12 + 20 + 28 = 60 > 50; =10

The Corresponding Ceria inequality is:

x1+x2+x3 ≤ 10

(4,4,2), (4,2,4) & (2,4,4) are 3 affinely independent points that satisfy this at equality. However only (4,4,2) and (4,2,4) are feasible… this constraint is not facet defining.

22

… but we can fix that with Strong minimal dependence.

• A Cover C is Strongly minimally dependant (SMD) if (additionally) for some k :

BUT

• The Ceria inequality is facet defining iff the cover C happens to be SMD

\{ }

+( -k) ,j j i ij C i

u a u a b i C

\{ }

+( -k+1) > ,j j i ij C i

u a u a b i C

• Should C not be SMD:

• Project out variables by fixing them to their upper bounds, until it is:

• 3x1+5x2+7x3 ≤ 50, 0 ≤ xi ≤4

• {1,2,3} is a cover, but not SMD.

• Project out x1:

• 5x2+7x3 ≤ 50-12=48, 0 ≤ xi ≤4 [l=10]

• {2,3} is SMD (k=2) for the reduced space.

• x2+x3 ≤6 is a facet for the reduced space [both (2,4) and (4,2) lie on it and are feasible)]

• By applying lifting to get a constraint for the original space:

• 0.5x1+x2+x3 ≤8

• [which is facet defining- (0,4,4), (4,2,4) & (4,4,2) lie on it]

• The SMD condition tells us when we need to project and then lift in order to get a facet

23

… and sometimes minimal coves are too small…

Problem:

Any minimal cover with:

cannot ever lead to a KCI that cuts off xlp, as

is a constraint in the current LP, and clearly:

i ii C

a xlp b

i ii C

a xlp b

i i i ii C i C

a xlp a xlp

… worse still- for some knapsack/LP solution combinations all minimal covers have this problem…

Example:

u=(6,4), xlp=(2,2.75)

6x1+4x2≤23

{1} is the only minimal cover. The corresponding KCI :

x1≤3 does not cut off the current LP solution.

24

… we can (often- but not always) fix that with Ceil(x) minimality

• Solution- Define a more complicated generalisation of minimality:

A cover is Ceil-x-minimal if

and

and

(Make the cover too big however and no SMD subcover exists)

- in binary spaces all ceil-x-minimal covers are also minimal covers so this issue does not arise.

- Occasionally no ceil-x-minimal cover exists- and therefore no violated KCI

i ii C

a xlp b

i ii S C

a xlp b

\{1}i i

i C

a u b

25

… and get significant computational improvements

log(CPU time (s))

0.01

0.1

1

10

100

1000

10000

100000

Nothing w / SMD/Ceil X Ceria

log(# nodes)

1

10

100

1000

10000

100000

Nothing w / SMD/Ceil X Ceria