Upload
corey-rogers
View
214
Download
2
Tags:
Embed Size (px)
Citation preview
Techniques for Computing Techniques for Computing and Using Bounds for and Using Bounds for
Combinatorial Combinatorial Optimization ProblemsOptimization Problems
Sharlee Climer and Weixiong Sharlee Climer and Weixiong ZhangZhang
Department of Computer Science and Department of Computer Science and EngineeringEngineering
Washington University in St. LouisWashington University in St. LouisThis research was funded in part by NDSEG and Olin Fellowships, NSF grants IIS-0196057
and ITR/EIA-0113618, and in part by DARPA Cooperative Agreement F30602-00-2-0531
IJCAI-05 TutorialIJCAI-05 Tutorial new slidenew slide
Complete slides will be available at:Complete slides will be available at:
www.cse.wustl.edu/~sclimerwww.cse.wustl.edu/~sclimer
or or
www.climer.uswww.climer.us
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 33
OverviewOverview
IntroductionIntroduction Formulation and notationFormulation and notation Historical perspectiveHistorical perspective Modifications to obtain boundsModifications to obtain bounds Two-step procedureTwo-step procedure Exploiting the use of boundsExploiting the use of bounds Future directionsFuture directions
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 44
What is the “use of bounds”?What is the “use of bounds”?
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 55
upper bound
optimalsolution
lower bound
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 66
upper bound
optimalsolution
lower bound
The use of boundsThe use of bounds
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 77
Use of boundsUse of bounds
Bounds have been extensively studied in Bounds have been extensively studied in both computer science and operations both computer science and operations research (OR)research (OR)
Pruning rules in branch-and-bound searchPruning rules in branch-and-bound search Without the use of bounds, many important Without the use of bounds, many important
problems would be unsolvableproblems would be unsolvable Previous efforts to systematically discover Previous efforts to systematically discover
effective effective relaxationsrelaxations to be used for bounds to be used for bounds Recent work to systematically discover Recent work to systematically discover
other modifications for use as boundsother modifications for use as bounds
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 88
Objectives of tutorialObjectives of tutorial
Survey the previous uses of bounds Survey the previous uses of bounds Reorganize existing work in a Reorganize existing work in a
systematic waysystematic way Point out potential directions for Point out potential directions for
future workfuture work
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 99
Formulation and notationFormulation and notation
Techniques presented can be applied Techniques presented can be applied to a variety of optimization problemsto a variety of optimization problems
To demonstrate the use of bounds, To demonstrate the use of bounds, we’ll use integer linear programs we’ll use integer linear programs (IPs) as basic problem structure(IPs) as basic problem structure
Without loss of generality, we Without loss of generality, we consider only minimization problemsconsider only minimization problems– Maximization problems can be cast as Maximization problems can be cast as
minimization problemsminimization problems
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 1010
Integer Linear ProgramsInteger Linear Programs
Minimize Minimize ZZ = = ccii x xi i (objective function)
Subject to: a set of linear constraintsSubject to: a set of linear constraints
xxii integer integer
If If xxii integer constraints omitted, would integer constraints omitted, would have a linear program (LP)have a linear program (LP)
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 1111
Integer Linear ProgramsInteger Linear Programs
Large number of commercial Large number of commercial applicationsapplications– SchedulingScheduling– RoutingRouting– PlanningPlanning
STRIPS planning problems converted STRIPS planning problems converted to IPs to IPs [Kautz and Walser, AAAI-99; Vossen et al. IJCAI-99][Kautz and Walser, AAAI-99; Vossen et al. IJCAI-99]
– Simplifies adding costs and resourcesSimplifies adding costs and resources– Optimality conditionsOptimality conditions
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 1212
Integer Linear ProgramsInteger Linear Programs Used to model:Used to model:
– Traveling Salesman Problem Traveling Salesman Problem – Constraint Satisfaction Problem Constraint Satisfaction Problem – Robotic motion problemsRobotic motion problems– Clustering Clustering – Multiple sequence alignment Multiple sequence alignment – Haplotype inferencingHaplotype inferencing– VLSI circuit designVLSI circuit design– Computer disk read head schedulingComputer disk read head scheduling– Derivation of physical structures of programsDerivation of physical structures of programs– Delay-Tolerant Network routing Delay-Tolerant Network routing – Cellular radio network base station locationsCellular radio network base station locations– Minimum-energy multicast problem in wireless ad hoc Minimum-energy multicast problem in wireless ad hoc
networks networks
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 1313
Linear program exampleLinear program example
Minimize Z = -11x + 4y
Subject to:3x + 8y <= 4011x - 8y <= 16x,y >= 0
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 1414
Linear program exampleLinear program example
Minimize Z = -11x + 4y
y = 11/4 x + Z/4
Family of parallel lines with slope of 11/4 and unknown y-intercept
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 1515
Linear program exampleLinear program example
Optimal solution x = 4y = 7/2Z = -30
Optimal solutionis always on a vertex
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 1616
Integer linear programInteger linear program
Minimize Z = -11x + 4y
Subject to:3x + 8y <= 4011x - 8y <= 16
x,y >= 0x,y integer
Optimal solution x = 3y = 3Z = -21
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 1717
Dual problemDual problem
Every IP has a Every IP has a dualdual problem that is problem that is also an IPalso an IP
Original IP referred to as Original IP referred to as primalprimal If primal is a minimization IP, then If primal is a minimization IP, then
dual is a maximization IPdual is a maximization IP Same coefficients, but rearrangedSame coefficients, but rearranged Inequalities in constraints are Inequalities in constraints are
reversedreversed
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 1818
Dual linear programDual linear program
Primal:
Minimize -11x + 4y
Subject to:-3x - 8y >= -40-11x + 8y >= -16x,y >= 0
Dual:
Maximize -40v - 16w
Subject to:-3v - 11w <= -11-8v + 8w <= 4v,w >= 0
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 1919
Dual linear programDual linear program
Maximize D = -40v - 16w
Subject to:-3v - 11w <= -11-8v + 8w <= 4v,w >= 0
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 2020
Dual linear programDual linear program
Optimal solution v = 11/28w = 25/28D = -30
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 2121
Dual integer programDual integer programMaximize D = -40v - 16w
Subject to:-3v - 11w <= -11-8v + 8w <= 4v,w >= 0v,w integer
Optimal solution v = 1w = 1D = -56
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 2222
The Traveling Salesman The Traveling Salesman ProblemProblem
The Traveling Salesman Problem The Traveling Salesman Problem (TSP) is the problem of finding a (TSP) is the problem of finding a minimum cost complete tour of a set minimum cost complete tour of a set of citiesof cities
STSP: cost from city STSP: cost from city i i to city to city jj is is equal to the cost from city equal to the cost from city jj to city to city ii
ATSP: costs not necessarily equalATSP: costs not necessarily equal
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 2323
The Traveling Salesman The Traveling Salesman ProblemProblem
Minimize Minimize ZZ = = ccijij x xij ij
s.t.: s.t.: xxij ij = 1 = 1 for for jj = 1,…, = 1,…,nn
xxij ij = 1 for = 1 for ii = 1,…, = 1,…,nn
xxij ij <= |<= |WW| - 1, for all proper | - 1, for all proper non- non- empty subsets empty subsets WW of of VV
xxijij = {0,1} = {0,1}
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 2424
Reprinted by permission, G. Dantzig, R. Fulkerson, and S. Johnson, Reprinted by permission, G. Dantzig, R. Fulkerson, and S. Johnson, Solution of a Large-Scale Solution of a Large-Scale Traveling-Salesman ProblemTraveling-Salesman Problem, Journal of the Operations Research Society of America, volume 2, , Journal of the Operations Research Society of America, volume 2,
1954. Copyright 1954, the Institute for Operations Research and the Management Sciences, 7240 1954. Copyright 1954, the Institute for Operations Research and the Management Sciences, 7240 Parkway Drive, Suite 310, Hanover, Maryland 21076.Parkway Drive, Suite 310, Hanover, Maryland 21076.
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 2525
Omit subtour elimination Omit subtour elimination constraintsconstraints
Minimize Minimize ZZ = = ccijij x xij ij
s.t.: s.t.: xxij ij = 1 = 1 for for jj = 1,…, = 1,…,nn
xxij ij = 1 for = 1 for ii = 1,…, = 1,…,nn
xxij ij <= |<= |WW| - 1, for all proper non- | - 1, for all proper non- empty subsets empty subsets WW of of VV
xxijij = {0,1} = {0,1}
The assignment problemThe assignment problem A lower bound for the ATSP A lower bound for the ATSP Can be solved in polynomial timeCan be solved in polynomial time
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 2626
Omit subtour elimination Omit subtour elimination constraintsconstraints
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 2727
Relax integrality constraintsRelax integrality constraints
Minimize Minimize ZZ = = ccijij x xij ij
s.t.: s.t.: xxij ij = 1 = 1 for for jj = 1,…, = 1,…,nn xxij ij = 1 for = 1 for ii = 1,…, = 1,…,nn xxij ij <= |<= |WW| - 1, for all proper non- | - 1, for all proper non-
empty subsets empty subsets WW of of VV xxijij = {0,1} = {0,1} 0 <= 0 <= xxijij <= 1 <= 1
Linear program (LP) relaxationLinear program (LP) relaxation Held-Karp lower bound Held-Karp lower bound Can be solved in polynomial timeCan be solved in polynomial time
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 2828
Relax integrality constraintsRelax integrality constraints
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 2929
Historical perspectiveHistorical perspective
Branch-and-boundBranch-and-bound Alpha-beta pruningAlpha-beta pruning Admissible heuristicsAdmissible heuristics
– Abstractions Abstractions – Pattern databasesPattern databases
Cutting planesCutting planes Gomory cutsGomory cuts Branch-and-cut Branch-and-cut
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 3030
Branch-and-boundBranch-and-bound
In 1958, several papers appeared using In 1958, several papers appeared using branch-and-bound (BnB) branch-and-bound (BnB) [Bock, Op. Res. 1958; [Bock, Op. Res. 1958; Croes, Op. Res. 1958; Eastman, PhD thesis 1958, Rossman Croes, Op. Res. 1958; Eastman, PhD thesis 1958, Rossman and Twery, Op. Res. 1958]and Twery, Op. Res. 1958]
Three of these papers introduced Three of these papers introduced algorithms for the TSPalgorithms for the TSP
““Branch-and-bound” coined Branch-and-bound” coined [Little et al. Op. Res. [Little et al. Op. Res.
1963]1963] (also a TSP algorithm) (also a TSP algorithm) Example: Carpaneto, Dell’Amico, and Example: Carpaneto, Dell’Amico, and
Toth’s (CDT) algorithm for the ATSP Toth’s (CDT) algorithm for the ATSP [ACM [ACM Trans. On Math. Software 1995]Trans. On Math. Software 1995]
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 3131
CDT algorithmCDT algorithm
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 3232
Branch-and-boundBranch-and-bound
Croes TSP algorithm perhaps first BnB Croes TSP algorithm perhaps first BnB search published search published [Croes, Op. Res. 1958][Croes, Op. Res. 1958]
Used bounds in two different waysUsed bounds in two different ways Found an approximate solutionFound an approximate solution Used solution as upper bound for Used solution as upper bound for
eliminating sets of arcs that cannot eliminating sets of arcs that cannot simultaneously appearsimultaneously appear
BnB tree over remaining arc BnB tree over remaining arc combinationscombinations
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 3333
Branch-and-boundBranch-and-bound
Reprinted by permission, G. A. Croes, Reprinted by permission, G. A. Croes, A Method for Solving Traveling-Salesman A Method for Solving Traveling-Salesman ProblemsProblems, Operations Research, volume 6, 1958. Copyright 1958, the Institute for , Operations Research, volume 6, 1958. Copyright 1958, the Institute for Operations Research and the Management Sciences, 7240 Parkway Drive, Suite 310, Operations Research and the Management Sciences, 7240 Parkway Drive, Suite 310, Hanover, Maryland 21076.Hanover, Maryland 21076.
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 3434
Branch-and-boundBranch-and-bound
Croes solved 49-city STSP that had Croes solved 49-city STSP that had previously been solved by Dantzig, previously been solved by Dantzig, Fulkerson, and Johnson Fulkerson, and Johnson [Op. Res. 1954][Op. Res. 1954]
Improvements:Improvements:– Mechanized solverMechanized solver– Provides “anytime” solutionProvides “anytime” solution– Faster than previous methodsFaster than previous methods
Croes solved by handCroes solved by hand– 70 hours to solve70 hours to solve– Found optimal solution after 10 hoursFound optimal solution after 10 hours
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 3535
Alpha-beta pruningAlpha-beta pruning
Minimax search is a specialized BnB Minimax search is a specialized BnB search for game playingsearch for game playing
Objective is to determine the best Objective is to determine the best movemove
Alpha-beta pruning used to reduce Alpha-beta pruning used to reduce the number of nodes that need to be the number of nodes that need to be evaluatedevaluated
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 3636
Alpha-beta pruningAlpha-beta pruning
Devised by John McCarthy in 1956, Devised by John McCarthy in 1956, but not publishedbut not published
Used in the NSS chess program in Used in the NSS chess program in 1958 1958 [Newell et al. IBM Journal of R&D 1958][Newell et al. IBM Journal of R&D 1958]
Pearl proved alpha-beta to be an Pearl proved alpha-beta to be an asymptotically optimal game-asymptotically optimal game-searching algorithm searching algorithm [Pearl, Comm. of ACM 1982][Pearl, Comm. of ACM 1982]
Modified strategy used by Deep Blue Modified strategy used by Deep Blue [Hsu, [Hsu, Behind Deep BlueBehind Deep Blue 2002] 2002]
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 3737
Minimax search treeMinimax search tree
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 3838
Alpha-beta pruningAlpha-beta pruning
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 3939
Alpha-beta pruningAlpha-beta pruning
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 4040
Alpha-beta pruningAlpha-beta pruning
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 4141
Alpha-beta pruningAlpha-beta pruning
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 4242
Alpha-beta pruningAlpha-beta pruning
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 4343
Alpha-beta pruningAlpha-beta pruning
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 4444
Admissible heuristicsAdmissible heuristics
Evaluation functions used by minimax Evaluation functions used by minimax are examples of are examples of heuristicsheuristics
Trade-off of accuracy vs. time to Trade-off of accuracy vs. time to computecompute
Admissible Admissible heuristic provides a lower heuristic provides a lower boundbound
Admissibility needed to guarantee A* Admissibility needed to guarantee A* search performancesearch performance
Admissibility allows pruningAdmissibility allows pruning
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 4545
Admissible heuristicsAdmissible heuristics
Abstractions used for admissible heuristics Abstractions used for admissible heuristics can be generated by relaxing constraints can be generated by relaxing constraints
In 1970, Held and Karp used relaxation for In 1970, Held and Karp used relaxation for the STSP the STSP [Op. Res. 1970][Op. Res. 1970] – 1-tree1-tree – A spanning tree with an added edgeA spanning tree with an added edge– Node Node 11 has degree two and is part of single has degree two and is part of single
cyclecycle– Find MST without node Find MST without node 11 and add two smallest and add two smallest
edges incident to node edges incident to node 11
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 4646
11-tree-tree
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 4747
Systematic generationSystematic generation
In 1982, Pearl suggested automatically In 1982, Pearl suggested automatically deriving admissible heuristics by deriving admissible heuristics by systematically deleting constraints systematically deleting constraints [Pearl, [Pearl, reprinted in AI Magazine 1983]reprinted in AI Magazine 1983]
Absolver II Absolver II [Prieditis, Machine Learning 1993][Prieditis, Machine Learning 1993]
– An implementation to automatically derive An implementation to automatically derive admissible heuristics admissible heuristics
– Uses Uses abstracting transformationsabstracting transformations Reduces cost function and/orReduces cost function and/or Expands goal statesExpands goal states
– Used to find first useful heuristic for Rubik’s cubeUsed to find first useful heuristic for Rubik’s cube* Introduced earlier by Somalvico et al.* Introduced earlier by Somalvico et al.
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 4848
Pearl’s examplePearl’s example
Used STRIPS formulation of 8-Puzzle Used STRIPS formulation of 8-Puzzle problemproblem
33 55 11
77 22
66 44 88
11 22
33 44 55
66 77 88
Start state Goal state
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 4949
Pearl’s examplePearl’s example
3 primitive predicates3 primitive predicates– ON(ON(x,yx,y) tile ) tile xx is on cell is on cell yy– CLEAR(CLEAR(yy) cell is clear of tiles) cell is clear of tiles– ADJ(ADJ(y,zy,z) cell ) cell yy is adjacent to cell is adjacent to cell zz
Each state defined by a list of 8 ON Each state defined by a list of 8 ON predicates, one CLEAR predicate, and predicates, one CLEAR predicate, and a fixed set of ADJ predicates a fixed set of ADJ predicates describing the adjacencies of the describing the adjacencies of the cells on the boardcells on the board
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 5050
Pearl’s examplePearl’s example
Operator to move tile Operator to move tile xx from cell from cell yy to to cell cell zzMOVE(MOVE(x,y,zx,y,z))Precondition list: ON(Precondition list: ON(x,yx,y), CLEAR(), CLEAR(zz), ADJ(), ADJ(y,zy,z))Add list: ON(Add list: ON(x,zx,z), CLEAR(), CLEAR(yy))Delete list: ON(Delete list: ON(x,yx,y), CLEAR(), CLEAR(zz))
Find a sequence of MOVE operations to Find a sequence of MOVE operations to transform from the initial state to a goal transform from the initial state to a goal statestate
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 5151
Pearl’s examplePearl’s example
Removing CLEAR(Removing CLEAR(zz) and ADJ() and ADJ(y,zy,z) from ) from the precondition list the precondition list – Permits a tile to be moved from its current Permits a tile to be moved from its current
position to any other position in one moveposition to any other position in one move– Equal to the number of tiles that are Equal to the number of tiles that are
misplaced in initial statemisplaced in initial state Removing CLEAR(Removing CLEAR(zz) from the ) from the
precondition listprecondition list– Equals the sum of the Manhattan Equals the sum of the Manhattan
distancesdistances
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 5252
Pearl’s examplePearl’s example Removing ADJ(Removing ADJ(y,zy,z) from the precondition list) from the precondition list Swap the blank with any other tileSwap the blank with any other tile Can be solved with greedy algorithmCan be solved with greedy algorithm
– If current empty cell If current empty cell yy is to be covered with tile is to be covered with tile xx, move , move x x to cell to cell y. y. Else move any misplaced Else move any misplaced tile into tile into yy..
Less intuitive than other two heuristicsLess intuitive than other two heuristics Discovered 13 years after A* was tested Discovered 13 years after A* was tested
using the other twousing the other two Constraints can be dividedConstraints can be divided
– Tighter lower bound if only part of constraint is Tighter lower bound if only part of constraint is removedremoved
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 5353
Pattern databasesPattern databases A look-up table containing precomputed A look-up table containing precomputed
solutions to subproblemssolutions to subproblems Cost of solving entire problem is at least Cost of solving entire problem is at least
as large as cost of solving subproblemas large as cost of solving subproblem Large memory requirementsLarge memory requirements Used to find first optimal solutions to Used to find first optimal solutions to
random Rubik’s Cube random Rubik’s Cube [Korf, Workshop on Computer [Korf, Workshop on Computer Games, IJCAI-97]Games, IJCAI-97]
Introduced by Culberson and Schaeffer Introduced by Culberson and Schaeffer and demonstrated on 15-puzzle and demonstrated on 15-puzzle [Lecture Notes [Lecture Notes in AI, 1996]in AI, 1996]
– Precomputed minimum cost for placing 7 tiles Precomputed minimum cost for placing 7 tiles plus the “empty” tile in correct final positionplus the “empty” tile in correct final position
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 5454
Pattern databasesPattern databases
bb 33
77
1111
1212 1313 1414 1515
bb
88 99 1010
1212 1313 1414 1515
Fringe target pattern Corner target pattern
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 5555
Cutting planesCutting planes OR research moved in different directionOR research moved in different direction Cutting planes: added constraints that Cutting planes: added constraints that
tighten a relaxationtighten a relaxation Introduced by Dantzig, Fulkerson, and Introduced by Dantzig, Fulkerson, and
Johnson Johnson [Op. Res. 1954][Op. Res. 1954]– 49-city STSP49-city STSP– Iteratively solve relaxationIteratively solve relaxation– Integrality and subtour elimination constraints Integrality and subtour elimination constraints
(SECs) removed(SECs) removed– Add cutting planes to remove relaxed solutionsAdd cutting planes to remove relaxed solutions– Added 23 SECs and 2 subjectively improvised Added 23 SECs and 2 subjectively improvised
cuts to remove non-integral solutions cuts to remove non-integral solutions
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 5656
Generate cutsGenerate cuts
General procedure to remove non-integral General procedure to remove non-integral solutionssolutions
from binary IPs, as outlined by Hillier and from binary IPs, as outlined by Hillier and LiebermanLieberman
[2001][2001]
1.1. Consider <= constraint with nonnegative Consider <= constraint with nonnegative coefficientscoefficients
2.2. Find a group of variables such thatFind a group of variables such that(a)(a) Violation if all equal 1 (while others equal 0)Violation if all equal 1 (while others equal 0)(b)(b) Satisfied if Satisfied if any oneany one is changed to 0 is changed to 0
3.3. Let Let k k equal number of variables in groupequal number of variables in groupSum of variables in group <= Sum of variables in group <= k k - 1- 1
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 5757
Example Example
1010xx + 16 + 16yy + 13 + 13zz <= 29 <= 29x, y, zx, y, z are binary variables are binary variablesIntegrality is relaxedIntegrality is relaxed
Constraint is violated if Constraint is violated if x, y, x, y, and and zz are all equal to 1 are all equal to 1Constraint is satisfied if any one is equal to 0Constraint is satisfied if any one is equal to 0New cut:New cut:xx + + yy + + zz <= 2 <= 2
Cut removes solutions from relaxed problem but not Cut removes solutions from relaxed problem but not from original problem (from original problem (e.g. xe.g. x = 0.5, = 0.5, yy = 0.8, = 0.8, zz = = 0.8)0.8)
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 5858
Gomory cutsGomory cuts
In 1958, Gomory proposed an iterative In 1958, Gomory proposed an iterative search strategy for IPs search strategy for IPs [Bulletin of the Am. Math. [Bulletin of the Am. Math. Soc., 1958]Soc., 1958]
– Solve with integrality relaxedSolve with integrality relaxed– Apply cut to remove relaxed solutionApply cut to remove relaxed solution– Proved terminationProved termination
In 1966, Martin implemented first TSP In 1966, Martin implemented first TSP code using cutting planes code using cutting planes [Op. Res. 1966][Op. Res. 1966]
– Relaxed integrality and subtour elimination Relaxed integrality and subtour elimination constraints (SECs)constraints (SECs)
Iterative procedures, no branchingIterative procedures, no branching
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 5959
Branch-and-cutBranch-and-cut
Branch-and-cut (BnC) is a BnB search Branch-and-cut (BnC) is a BnB search with cutting planes added to with cutting planes added to relaxations at nodesrelaxations at nodes
Hong implemented BnC code in 1972 Hong implemented BnC code in 1972 for the TSP for the TSP [PhD thesis, Johns Hopkins, 1972][PhD thesis, Johns Hopkins, 1972]
In 1987, Padberg and Rinaldi coined In 1987, Padberg and Rinaldi coined “branch-and-cut” and used to solve “branch-and-cut” and used to solve 532-city STSP 532-city STSP [Op. Res. Letters 1987][Op. Res. Letters 1987]
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 6060
ConcordeConcorde In 1994, first implementation of Concorde In 1994, first implementation of Concorde
[Applegate et al. Lecture Notes in Comp. Sci. 2001][Applegate et al. Lecture Notes in Comp. Sci. 2001]
Branch-and-cut code for STSPBranch-and-cut code for STSP Relaxed integrality and SECsRelaxed integrality and SECs Also used custom cuts tailored for the TSPAlso used custom cuts tailored for the TSP Apply cuts at a node until diminishing returnsApply cuts at a node until diminishing returns Branch by setting value of an arcBranch by setting value of an arc In 2004, Concorde was used to solve a In 2004, Concorde was used to solve a
24,978-city STSP 24,978-city STSP [Applegate et al. [Applegate et al. www.tsp.gatech.eduwww.tsp.gatech.edu]]
– 14,827,429 cutting planes in addition to SECs14,827,429 cutting planes in addition to SECs
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 6161
Modifications to obtain Modifications to obtain boundsbounds
Many possibilities for obtaining bounds Many possibilities for obtaining bounds have been previously overlookedhave been previously overlooked
Examine every aspect of problem Examine every aspect of problem descriptiondescription
Modifications of IPs to produce boundsModifications of IPs to produce bounds– Relaxing or tightening constraintsRelaxing or tightening constraints– Modifying objective functionModifying objective function– Adding or deleting decision variablesAdding or deleting decision variables– Modifications to dual problemModifications to dual problem
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 6262
ExampleExample
Minimize Minimize ZZ = = yy –– 4/5 4/5 x x
Subject to: Subject to: x x >= 0>= 0y y <=<= 33y y + 13/6+ 13/6 x x <=<= 99y – y – 5/135/13 x x >= 1/14>= 1/14y y ++ 3/53/5 x x >=>= 6/56/5
x,yx,y integers integers
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 6363
Solution spaceSolution space
x >= 0y <= 3y + 13/6 x <= 9y – 5/13 x >= 1/14y + 3/5 x >= 6/5x,y integers
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 6464
Objective functionObjective function
Minimize Z = y – 4/5 x
x = 0y = 3Z = 3
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 6565
Optimal solution Optimal solution
Minimize Z = y – 4/5 x
x = 2y = 1Z = -0.6
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 6666
Relaxing constraintsRelaxing constraints
Minimize Z = y – 4/5 xsubject to:x >= 0y <= 3y + 13/6 x <= 9y – 5/13 x >= 1/14y + 3/5 x >= 6/5x,y integers
x = 3.5y = 1.4Z = -1.4
Lower bound
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 6767
Relaxing constraints Relaxing constraints
Minimize Z = y – 4/5 xsubject to:x >= 0y <= 3y + 13/6 x <= 9y – 5/13 x >= 1/14y + 3/5 x >= 6/5x,y integers
x = 3y = 1
Z = -1.4Lower bound
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 6868
Tightening constraintsTightening constraints
Minimize Z = y – 4/5 xsubject to:x >= 0y <= 3y + 13/6 x <= 9y – 5/13 x >= 1/14y + 3/5 x >= 6/5x,y integersy >= 2
x = 3y = 2
Z = -0.4Upper bound
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 6969
Tightening constraintsTightening constraints
Minimize Z = y – 4/5 xsubject to:x >= 0y <= 3y + 13/6 x <= 9y – 5/13 x >= 1/14y + 3/5 x >= 6/5x,y integers
x = 2y = 2
Z = 0.4Upper bound
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 7070
Tightening constraintsTightening constraints
Common use: adding constraints that Common use: adding constraints that set values of variables (branching set values of variables (branching rules)rules)
Reduce number of feasible solutionsReduce number of feasible solutions Example of “easier” problem: add Example of “easier” problem: add
constraints setting most variables to constraints setting most variables to zero, and solve sparse problemzero, and solve sparse problem
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 7171
Relaxing optimalityRelaxing optimality
Minimize Z = y – 4/5 xsubject to:x >= 0y <= 3y + 13/6 x <= 9y – 5/13 x >= 1/14y + 3/5 x >= 6/5x,y integers
x = 3y = 2
Z = -0.4Upper bound
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 7272
Modifying objective function Modifying objective function coefficientscoefficients
Minimize Z = y – 4/5 xsubject to:x >= 0y <= 3y + 13/6 x <= 9y – 5/13 x >= 1/14y + 3/5 x >= 6/5x,y integers
x = 1y = 1Z = 1
Upper bound
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 7373
Modifying objective function Modifying objective function coefficientscoefficients
Minimize Z = y – 4/5 xsubject to:x >= 0y <= 3y + 13/6 x <= 9y – 5/13 x >= 1/14y + 3/5 x >= 6/5x,y integers
x = 1y = 1
Z = 0.2Upper bound
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 7474
Modifying objective function Modifying objective function coefficientscoefficients
Reducing range of objective function Reducing range of objective function coefficients can yield “easier” problem coefficients can yield “easier” problem for TSPfor TSP
Phase transitionsPhase transitions – difficulty in solving – difficulty in solving problem changes dramatically when a problem changes dramatically when a parameter is increased beyond a parameter is increased beyond a distinct valuedistinct value
For TSP, parameter is range of For TSP, parameter is range of ccijij valuesvalues
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 7575
Modifying objective function Modifying objective function coefficientscoefficients
Zhang reduced range by eliminating Zhang reduced range by eliminating least significant bits least significant bits [CP-AI-OR-02][CP-AI-OR-02]
After finding solution, used original After finding solution, used original values to compute cost of tour, values to compute cost of tour, yielding an upper boundyielding an upper bound
Could derive a lower bound by Could derive a lower bound by rounding values down and keeping rounding values down and keeping these values when computing cost of these values when computing cost of tour foundtour found
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 7676
Modifying objective function Modifying objective function coefficientscoefficients
Frieze’s polynomial-time STSP Frieze’s polynomial-time STSP algorithm algorithm [SIAM Computing 1987][SIAM Computing 1987]
– Finds Finds exactexact solution with a probability solution with a probability that tends to 1 as the number of cities that tends to 1 as the number of cities nn tends to infinitytends to infinity
– Random cost values drawn from zero to Random cost values drawn from zero to
B B ((nn) - 1, where ) - 1, where B B ((nn) = ) = oo ( (nn / log log / log log nn))
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 7777
Modifying objective function Modifying objective function coefficientscoefficients
Lagrangian relaxationLagrangian relaxation– Adds penalties to objective function for Adds penalties to objective function for
violation of deleted constraints violation of deleted constraints [Fisher, [Fisher, Management Science 1981]Management Science 1981]
– Deletion of constraints yields lower boundDeletion of constraints yields lower bound– Addition of penalties tightens the bound, Addition of penalties tightens the bound,
but it is still a lower boundbut it is still a lower bound– Can be used as a lower bound for pruning Can be used as a lower bound for pruning
or as a heuristic to guide the searchor as a heuristic to guide the search
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 7878
Adding decision variablesAdding decision variables
Assuming zero is in allowable range for Assuming zero is in allowable range for new variablenew variable
Addition of variable creates a lower Addition of variable creates a lower boundbound
Creates additional feasible solutions, Creates additional feasible solutions, doesn’t exclude any feasible solutions to doesn’t exclude any feasible solutions to original problemoriginal problem
Example 1: add edges in graph problemsExample 1: add edges in graph problems– May create shortcutsMay create shortcuts
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 7979
Adding decision variablesAdding decision variables
Example 2: tilted drilling machine Example 2: tilted drilling machine problems, a class of ATSPsproblems, a class of ATSPs
Convert ATSP to STSP with 2-node Convert ATSP to STSP with 2-node transformationtransformation
Solved 800 100-city instances using Solved 800 100-city instances using Concorde and ATSP solversConcorde and ATSP solvers
Number of decision variables more than Number of decision variables more than quadrupled*, yet modified problems were quadrupled*, yet modified problems were generally easier to solvegenerally easier to solve
*number of variables without zero or infinite costs same as original problem
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 8080
Deleting decision variablesDeleting decision variables Assuming zero is in allowable range for Assuming zero is in allowable range for
deleted variabledeleted variable Deletion of variable creates an upper boundDeletion of variable creates an upper bound Removes feasible solutions from the Removes feasible solutions from the
original problemoriginal problem Example for TSP: remove all arcs with Example for TSP: remove all arcs with
edges greater than a thresholdedges greater than a threshold– If a tour exists it is an upper boundIf a tour exists it is an upper bound
Same result as adding a constraint setting Same result as adding a constraint setting the value of the variable to zero the value of the variable to zero
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 8181
Using dual problemUsing dual problem Feasible solutions to dual IP are lower Feasible solutions to dual IP are lower
bounds to primal IPbounds to primal IP Modifications to dual that yield lower Modifications to dual that yield lower
bounds for primalbounds for primal– Tightening constraints Tightening constraints – Relaxing optimality Relaxing optimality – Decreasing objective function coefficientsDecreasing objective function coefficients– Changing objective function coefficients, and Changing objective function coefficients, and
substituting original coefficients after dual is substituting original coefficients after dual is solvedsolved
– Deleting decision variables (assuming zero is in Deleting decision variables (assuming zero is in allowable range)allowable range)
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 8282
Summary of modificationsSummary of modificationsUpper Upper boundbound
Lower Lower boundbound
Relax constraintsRelax constraints -- PrimalPrimal
Tighten constraintsTighten constraints PrimalPrimal DualDual
Relax optimalityRelax optimality PrimalPrimal DualDual
Increase Increase ccii PrimalPrimal --
Decrease Decrease ccii -- Primal & dualPrimal & dual
Change Change ccii but use but use original values after original values after solvedsolved
PrimalPrimal DualDual
Delete variables*Delete variables* PrimalPrimal DualDual
Add variables*Add variables* -- PrimalPrimal
* Assuming zero is in allowable range of variables
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 8383
““Simplifying” modificationsSimplifying” modifications DecomposabilityDecomposability
– Manhattan distance for 8-puzzle exampleManhattan distance for 8-puzzle example– All subproblems can be solved independentlyAll subproblems can be solved independently
Partial orderingPartial ordering– 8-puzzle: delete ADJ requirement8-puzzle: delete ADJ requirement– Placing empty tile has higher order and is done Placing empty tile has higher order and is done
lastlast– Greedy algorithm worksGreedy algorithm works
Create special structuresCreate special structures– OR development of algorithms for tackling OR development of algorithms for tackling
special structures special structures Mutually exclusive alternatives constraintsMutually exclusive alternatives constraints Contingent decisions constraintsContingent decisions constraints Set-covering constraintsSet-covering constraints
Trial-and-errorTrial-and-error
IJCAI-05 TutorialIJCAI-05 Tutorial new slidenew slide
Set covering problemsSet covering problems
Given a set Given a set SS = {1, 2, 3, …, = {1, 2, 3, …, mm} and a } and a class class C C of subsets of of subsets of S S where each where each subset has a cost associated with itsubset has a cost associated with it
Cover all members of Cover all members of SS at minimum at minimum cost using members of cost using members of CC
Let Let SS = {1, 2, 3, 4, 5} and = {1, 2, 3, 4, 5} and CC = {{1, = {{1, 2}, {1, 3, 5}, {2, 4, 5}, {3}, {1}, {4, 2}, {1, 3, 5}, {2, 4, 5}, {3}, {1}, {4, 5}} with costs of 2, 1, 1, 2, 3, 1 5}} with costs of 2, 1, 1, 2, 3, 1 respectivelyrespectively
IJCAI-05 TutorialIJCAI-05 Tutorial new slidenew slide
Set covering problemsSet covering problems Let Let xxii = 1 if the = 1 if the i i thth member of member of CC is in the is in the
cover, otherwise cover, otherwise xxii = 0= 0
Minimize 2Minimize 2xx11 + + xx22 + + xx33 + 2+ 2xx44 + 3+ 3xx55 + + xx66
Subject to:Subject to: xx11 + + xx22 + + xx55 >= 1>= 1xx11 + + xx33 >= 1>= 1 + + xx2 2 + + xx44 >= 1>= 1 + + xx33 + + xx66 >= 1>= 1 + + xx22 + + xx33 + + xx66 >= 1>= 1
IJCAI-05 TutorialIJCAI-05 Tutorial new slidenew slide
Set covering problemsSet covering problems
Set covering propertiesSet covering properties– Minimization with all constraints ‘>=‘Minimization with all constraints ‘>=‘– All RHS coefficients are 1All RHS coefficients are 1– All other matrix coefficients are 0 or 1All other matrix coefficients are 0 or 1
WeightedWeighted set covering set covering – RHS >= 1RHS >= 1
GeneralizedGeneralized set covering set covering– RHS >= 1 RHS >= 1 – other matrix coefficients to be -1, 0, or 1other matrix coefficients to be -1, 0, or 1
IJCAI-05 TutorialIJCAI-05 Tutorial new slidenew slide
Set covering problemsSet covering problems
Comparatively easy to solveComparatively easy to solve Optimal solution must be a vertex Optimal solution must be a vertex
solution of the corresponding LPsolution of the corresponding LP However, this vertex solution is not However, this vertex solution is not
necessarily the optimal solution of necessarily the optimal solution of the LPthe LP
IJCAI-05 TutorialIJCAI-05 Tutorial new slidenew slide
Set packing problemsSet packing problems
Given a set Given a set SS = {1, 2, 3, …, = {1, 2, 3, …, mm} and a } and a class class C C of subsets of of subsets of S S where each where each subset has a value associated with itsubset has a value associated with it
Pack as many members of Pack as many members of C C as as possible into possible into SS yielding maximum yielding maximum total value, without overlapstotal value, without overlaps
IJCAI-05 TutorialIJCAI-05 Tutorial new slidenew slide
Set packing problemsSet packing problems Let Let xxii = 1 if the = 1 if the i i thth member of member of CC is in the is in the
pack, otherwise pack, otherwise xxii = 0= 0
Maximize 2Maximize 2xx11 + + xx22 + + xx33 + 2+ 2xx44 + 3+ 3xx55 + + xx66
Subject to:Subject to: xx11 + + xx22 + + xx55 <= 1<= 1xx11 + + xx33 <= 1<= 1 + + xx2 2 + + xx44 <= 1<= 1 + + xx33 + + xx66 <= 1<= 1 + + xx22 + + xx33 + + xx66 <= 1<= 1
IJCAI-05 TutorialIJCAI-05 Tutorial new slidenew slide
Set packing problemsSet packing problems
Set packing propertiesSet packing properties– Maximization with all constraints ‘<=‘Maximization with all constraints ‘<=‘– All RHS coefficients are 1All RHS coefficients are 1– All other matrix coefficients are 0 or 1All other matrix coefficients are 0 or 1
WeightedWeighted set packing set packing – RHS >= 1RHS >= 1
GeneralizedGeneralized set packing set packing– RHS >= 1 RHS >= 1 – other matrix coefficients to be -1, 0, or 1other matrix coefficients to be -1, 0, or 1
IJCAI-05 TutorialIJCAI-05 Tutorial new slidenew slide
Set partitioning problemsSet partitioning problems
Given a set Given a set SS = {1, 2, 3, …, = {1, 2, 3, …, mm} and a } and a class class C C of subsets of of subsets of S S where each where each subset has a value associated with itsubset has a value associated with it
Pack as many members of Pack as many members of C C as as possible into possible into SS without overlaps without overlaps andand cover all the members of cover all the members of SS
Maximization or minimizationMaximization or minimization
IJCAI-05 TutorialIJCAI-05 Tutorial new slidenew slide
Set partitioning problemsSet partitioning problems Let Let xxii = 1 if the = 1 if the i i thth member of member of CC is in the is in the
pack, otherwise pack, otherwise xxii = 0= 0
Max or min 2Max or min 2xx11 + + xx22 + + xx33 + 2+ 2xx44 + 3+ 3xx55 + + xx66
Subject to:Subject to: xx11 + + xx22 + + xx55 = 1= 1xx11 + + xx33 = 1= 1 + + xx2 2 + + xx44 = 1= 1 + + xx33 + + xx66 = 1= 1 + + xx22 + + xx33 + + xx66 = 1= 1
IJCAI-05 TutorialIJCAI-05 Tutorial new slidenew slide
Set covering, packing, and Set covering, packing, and partitioning problemspartitioning problems
Set partitioning problems are equivalent to Set partitioning problems are equivalent to set packing problems set packing problems [Williams, [Williams, Model Building in Model Building in Mathematical ProgrammingMathematical Programming, 1985], 1985]
The optimal solution for either problem is a The optimal solution for either problem is a vertex solution for its corresponding LPvertex solution for its corresponding LP
These problems are generally even easier These problems are generally even easier than set covering problemsthan set covering problems– Corresponding LP is more constrained due to ‘=‘ Corresponding LP is more constrained due to ‘=‘
constraintsconstraints Problems can be modified to fully or Problems can be modified to fully or
partially resemble any of these problem partially resemble any of these problem typestypes
IJCAI-05 TutorialIJCAI-05 Tutorial new slidenew slide
Total unimodularityTotal unimodularity
IJCAI-05 TutorialIJCAI-05 Tutorial new slidenew slide
Total unimodularityTotal unimodularity
Convex hull of feasible integer points
IJCAI-05 TutorialIJCAI-05 Tutorial new slidenew slide
Total unimodularityTotal unimodularity
Deriving the convex hull is generally much Deriving the convex hull is generally much more difficult than solving the problemmore difficult than solving the problem
Sometimes the IP model is the convex hullSometimes the IP model is the convex hull– The problem is The problem is totally unimodulartotally unimodular– Assignment problemAssignment problem– Recognize and solve as LP instead of IP Recognize and solve as LP instead of IP
Sometimes problems can be easily Sometimes problems can be easily reformulated to yield the convex hullreformulated to yield the convex hull
Sometimes it is possible to reformulate to Sometimes it is possible to reformulate to get closer to the convex hullget closer to the convex hull
IJCAI-05 TutorialIJCAI-05 Tutorial new slidenew slide
Total unimodularityTotal unimodularity Let Let AA = the matrix consisting of the = the matrix consisting of the
coefficients of the decision variables in the coefficients of the decision variables in the constraints (not including the RHS)constraints (not including the RHS)
Reformulate problem so that it is a Reformulate problem so that it is a maximization problem, the values in maximization problem, the values in AA are are all non-negative, and the constraints are all non-negative, and the constraints are equalities (add equalities (add slack slack variables)variables)
AA is totally unimodular if every square is totally unimodular if every square submatrix of submatrix of AA has a determinant equal to has a determinant equal to -1, 0, or 1 -1, 0, or 1 [Garfinkel and Nemhauser, [Garfinkel and Nemhauser, Integer ProgrammingInteger Programming, , 1972]1972]
However, evaluating every determinant is However, evaluating every determinant is usually too difficultusually too difficult
IJCAI-05 TutorialIJCAI-05 Tutorial new slidenew slide
Total unimodularityTotal unimodularity
A sufficient but not necessary property A sufficient but not necessary property PP
Property Property PP : :– All values in All values in AA are -1, 0, or 1 are -1, 0, or 1– No more than 2 non-zero elements appear in No more than 2 non-zero elements appear in
each columneach column– Rows can be partitioned into 2 subsets such Rows can be partitioned into 2 subsets such
thatthat If a column contains 2 non-zero values with same If a column contains 2 non-zero values with same
sign, those rows are in different subsetssign, those rows are in different subsets If a column contains 2 non-zero values with different If a column contains 2 non-zero values with different
signs, those rows are in the same subsetsigns, those rows are in the same subset
IJCAI-05 TutorialIJCAI-05 Tutorial new slidenew slide
Total unimodularityTotal unimodularity
Easy to identify when property Easy to identify when property PP holdsholds
Sometimes possible to reformulate Sometimes possible to reformulate (or modify) problem so that property (or modify) problem so that property PP holds for the entire problem or for holds for the entire problem or for part of the problempart of the problem
IJCAI-05 TutorialIJCAI-05 Tutorial new slidenew slide
Total unimodularity Total unimodularity exampleexample
Common constraint:Common constraint:If any of a set of boolean If any of a set of boolean xxii are equal to 1, then are equal to 1, then xx
= 1= 1 Equivalent constraint:Equivalent constraint:
xx11 + + xx22 + … ++ … +xxnn - - nx nx <= 0<= 0 Reformulation into Reformulation into nn constraints: constraints:
xx11 - - x x <= 0<= 0......
xxnn - - x x <= 0<= 0
IJCAI-05 TutorialIJCAI-05 Tutorial new slidenew slide
Total unimodularity Total unimodularity exampleexample
Dual of reformulation has property Dual of reformulation has property PP Example also demonstrates another way to Example also demonstrates another way to
tighten a problemtighten a problem– IP is equivalent with or without reformulationIP is equivalent with or without reformulation– Sum of reformulated constraints equals the original Sum of reformulated constraints equals the original
constraintconstraint– Although IP is equivalent before and after reformulation, Although IP is equivalent before and after reformulation,
corresponding LPs are very differentcorresponding LPs are very different– Adding together a set of constraints in an LP generally Adding together a set of constraints in an LP generally
enlarges the feasible regionenlarges the feasible region– Fractional solutions to the LP for the original problem are Fractional solutions to the LP for the original problem are
ruled out in the LP of the reformulated problemruled out in the LP of the reformulated problem Modifications can be made to resemble total Modifications can be made to resemble total
unimodularity or to replace a single constraint unimodularity or to replace a single constraint with a set of constraint.with a set of constraint.
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 102102
Limit crossingLimit crossing
A 2-step procedure for systematically A 2-step procedure for systematically exploring the use of boundsexploring the use of bounds
Has been implicitly used in a number of Has been implicitly used in a number of algorithms and search strategiesalgorithms and search strategies
To our knowledge, hasn’t been To our knowledge, hasn’t been generally formalizedgenerally formalized– Narrower formalization Narrower formalization [Climer and Zhang, AAAI-02][Climer and Zhang, AAAI-02]
Broaden focus beyond branch-and-Broaden focus beyond branch-and-boundbound
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 103103
Limit crossingLimit crossing 2 steps:2 steps:
(1) Find a simple upper or lower bound(1) Find a simple upper or lower bound(2) Combine upper-bounding and lower- (2) Combine upper-bounding and lower-
bounding modifications and solve bounding modifications and solve
If solution of the doubly-modified problem If solution of the doubly-modified problem exceeds the simple upper bound, upper-exceeds the simple upper bound, upper-bounding modification in step (2) is invalidbounding modification in step (2) is invalid
If solution of doubly-modified problem is If solution of doubly-modified problem is less than the simple lower bound, lower-less than the simple lower bound, lower-bounding modification in step (2) is invalidbounding modification in step (2) is invalid
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 104104
Branch-and-bound searchBranch-and-bound search
Incumbentsolution
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 105105
Limit crossingLimit crossing
Find a simple upper or lower bound Find a simple upper or lower bound that is tightthat is tight
Systematically apply modifications to Systematically apply modifications to produce doubly-modified problemsproduce doubly-modified problems– Either modification can be difficult to Either modification can be difficult to
solvesolve– Only need the Only need the combinationcombination of the two of the two
modifications to be relatively easymodifications to be relatively easy May produce novel search strategiesMay produce novel search strategies
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 106106
Limit crossing search Limit crossing search strategiesstrategies
Cut-and-solve Cut-and-solve [Climer and Zhang, AIJ, to appear][Climer and Zhang, AIJ, to appear]
– An iterative search strategyAn iterative search strategy– Useful for combinatorial optimization Useful for combinatorial optimization
problemsproblems Backbone and fat identifier Backbone and fat identifier [Climer and [Climer and
Zhang, AAAI-02]Zhang, AAAI-02]
– Used to identify characteristic variablesUsed to identify characteristic variables
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 107107
Cut-and-solveCut-and-solve
Iterative search strategy called Iterative search strategy called cut-and-cut-and-solvesolve
For each iteration:For each iteration:– Step 1: A chunk of the solution space is cut Step 1: A chunk of the solution space is cut
away and solved, providing incumbent away and solved, providing incumbent solutionssolutions
– Step 2: A relaxed solution is found for Step 2: A relaxed solution is found for remaining solution spaceremaining solution space
Iterate until relaxed solution is greater Iterate until relaxed solution is greater than or equal to incumbentthan or equal to incumbent
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 108108
ExampleExample
x >= 0y <= 3y + 13/6 x <= 9y – 5/13 x >= 1/14y + 3/5 x >= 6/5x,y integers
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 109109
Optimal solution Optimal solution
Minimize Z = y – 4/5 x
x = 2y = 1Z = -0.6
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 110110
Iteration 1, first step Iteration 1, first step
Cut away a chunkof the solution space:
y – 17/3 x >= -14
and solve sparse problem
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 111111
Iteration 1, first step Iteration 1, first step
Incumbent solutionis -0.4
x = 3y = 2
Z = -0.4
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 112112
Iteration 1, second step Iteration 1, second step
Add new constraint:y – 17/3 x <= -14to cut off chunk of solution space
Relax integralityand solve
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 113113
Iteration 1, second step Iteration 1, second step
x = 2.6y = 1.0Z = -1.1
Incumbent solutionis -0.4, so need to run another iteration
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 114114
Iteration 2, first step Iteration 2, first step
Cut away a chunkof the solution spaceand solve sparse problem
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 115115
Iteration 2, first step Iteration 2, first step
This solution is less thanincumbent, so incumbentbecomes -0.6
x = 2y = 1
Z = -0.6
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 116116
Iteration 2, second stepIteration 2, second step
Add constraint to cutoff solved chunk
Relax integralityand solve
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 117117
Iteration 2, second stepIteration 2, second step
Incumbent value:Z = -0.6
Solution is greater than incumbent, so incumbentmust be optimal
x = 1.0y = 0.6Z = -0.2
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 118118
Cut-and-solve Cut-and-solve propertiesproperties
Minimal memory requirementsMinimal memory requirements– Keep new constraints and incumbent Keep new constraints and incumbent
solution from one iteration to the nextsolution from one iteration to the next No subtrees in which to get lostNo subtrees in which to get lost Piercing cuts should try to capture Piercing cuts should try to capture
optimal solutionsoptimal solutions Different from conventional cutting Different from conventional cutting
planesplanes– Remove solutions from original problemRemove solutions from original problem
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 119119
Cut-and-solveCut-and-solve
Same as two steps of limit crossingSame as two steps of limit crossing– Small chunk is solved to provide simple Small chunk is solved to provide simple
upper boundupper bound– Doubly-modified problemDoubly-modified problem
Added piercing cuts are upper-bounding Added piercing cuts are upper-bounding modificationsmodifications
Relaxation is lower-bounding modificationRelaxation is lower-bounding modification
Unusual upper-bounding modificationUnusual upper-bounding modification– Results in search path instead of search Results in search path instead of search
treetree
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 120120
Cut-and-solveCut-and-solve A general technique for determining A general technique for determining
piercing cuts for binary IPspiercing cuts for binary IPs– Relax integrality and solve LPRelax integrality and solve LP– LP yields LP yields reduced costsreduced costs
a lower bound on the increase of the solution if a lower bound on the increase of the solution if variable forced into LP solutionvariable forced into LP solution
– Let Let SS = set of = set of kk variables with the smallest variables with the smallest reduced costsreduced costs
– Find optimal solution for set Find optimal solution for set SS– Add piercing cut: sum of variables not in Add piercing cut: sum of variables not in SS >= >=
11 Custom piercing cuts tailored to particular Custom piercing cuts tailored to particular
problem may work betterproblem may work better
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 121121
Cut-and-solveCut-and-solve
We used general technique for ATSP We used general technique for ATSP [AIJ, [AIJ, to appear]to appear]
– Piercing cut added: sum of variables in Piercing cut added: sum of variables in SS <= <= nn – 1 – 1
– Same cut but less variables in constraintSame cut but less variables in constraint Very good results for difficult instancesVery good results for difficult instances
– Real-world instances with structural Real-world instances with structural characteristicscharacteristics
Results for easy problems not as Results for easy problems not as impressiveimpressive
IJCAI-05 TutorialIJCAI-05 Tutorial new slidenew slide
Real-world ATSPsReal-world ATSPs
Super Super - shortest common superstring- shortest common superstring Shop Shop - no-wait flowshop- no-wait flowshop Rtilt, stilt Rtilt, stilt - tilted drilling machine - tilted drilling machine Crane Crane - stacker crane- stacker crane Disk Disk - computer disk read head- computer disk read head Coin Coin - pay phone collection - pay phone collection
IJCAI-05 TutorialIJCAI-05 Tutorial new slidenew slide
ATSP resultsATSP results
100 city instances100 city instances Varied degree of accuracy of the arc Varied degree of accuracy of the arc
costscosts– Varied the number of digits used by Varied the number of digits used by
generatorsgenerators Average computation time over 100 Average computation time over 100
trialstrials Compared CDT, Concorde, and cut-Compared CDT, Concorde, and cut-
and-solveand-solve
IJCAI-05 TutorialIJCAI-05 Tutorial new slidenew slide
ATSP resultsATSP results
Super Super class not dependent on class not dependent on number of digitsnumber of digits
CDT averaged 0.073 secondsCDT averaged 0.073 seconds Concorde averaged 8.15 secondsConcorde averaged 8.15 seconds Cut-and-solve averaged 2.07 secondsCut-and-solve averaged 2.07 seconds
IJCAI-05 TutorialIJCAI-05 Tutorial new slidenew slide
ATSP resultsATSP results
shop class
IJCAI-05 TutorialIJCAI-05 Tutorial new slidenew slide
ATSP resultsATSP results
rtilt class
100 cities100 trials
IJCAI-05 TutorialIJCAI-05 Tutorial new slidenew slide
ATSP resultsATSP results
stilt class
IJCAI-05 TutorialIJCAI-05 Tutorial new slidenew slide
ATSP resultsATSP results
crane class
IJCAI-05 TutorialIJCAI-05 Tutorial new slidenew slide
ATSP resultsATSP results
disk class
IJCAI-05 TutorialIJCAI-05 Tutorial new slidenew slide
ATSP resultsATSP results
coin class
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 131131
Backbone and fat identifierBackbone and fat identifier
BackbonesBackbones are variables that appear are variables that appear in every optimal solutionin every optimal solution
FatFat variables don’t appear in any variables don’t appear in any optimal solutionoptimal solution
Useful for Useful for – Enumerating all optimal solutionsEnumerating all optimal solutions– Identifying critical componentsIdentifying critical components– Identifying wasted componentsIdentifying wasted components
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 132132
Backbone and fat identifierBackbone and fat identifier
Find a simple upper boundFind a simple upper bound Attempt to identify backbones:Attempt to identify backbones:
– Force exclusion of each variableForce exclusion of each variable– If limits cross, must be a backboneIf limits cross, must be a backbone
Attempt to identify fat:Attempt to identify fat:– Force inclusion of each variableForce inclusion of each variable– If limits cross, must be fatIf limits cross, must be fat
Use problem-specific information to Use problem-specific information to improve resultsimprove results
Example for ATSP Example for ATSP [Climer and Zhang, AAAI-02][Climer and Zhang, AAAI-02]
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 133133
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 134134
Upper bound = 61
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 135135
Upper bound = 61
Lower bound = 24
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 136136
Upper bound = 61
Lower bound = 24
Doubly-modified = 61
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 137137
Upper bound = 61
Lower bound = 24
Doubly-modified = 70
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 138138
Upper bound = 61
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 139139
Upper bound = 61
Lower bound = 24
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 140140
Upper bound = 61
Doubly-modified = 61
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 141141
Upper bound = 61
Doubly-modified = 75
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 142142
Upper bound = 61
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 143143
Upper bound = 61
Doubly-modified = 152
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 144144
Upper bound = 61
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 145145
Upper bound = 61
Doubly-modified = 115
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 146146
Upper bound = 61
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 147147
Upper bound = 61
Doubly-modified = infinity
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 148148
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 149149
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 150150
ResultsResults
Used AP lower bound Used AP lower bound [Climer and Zhang, AAAI-02][Climer and Zhang, AAAI-02] Random ATSP graphs with cost range Random ATSP graphs with cost range
equal to number of citiesequal to number of cities Found close to half of the backbone arcs Found close to half of the backbone arcs
and 99% of the fat arcsand 99% of the fat arcs Time savings when finding all solutions:Time savings when finding all solutions:
– 20% for smaller instances20% for smaller instances– 75% for larger instances75% for larger instances
Found very few backbones and fat for real-Found very few backbones and fat for real-world problems as this lower bound is not world problems as this lower bound is not tighttight– Implementing with tighter lower boundImplementing with tighter lower bound
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 151151
Moving beyond typical Moving beyond typical branch-and-boundbranch-and-bound
Cut-and-solve Backbone & fat identifier
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 152152
Future directionsFuture directions
Forecasting effective bounding modificationsForecasting effective bounding modifications– Automatically identifying “easy” doubly-modified Automatically identifying “easy” doubly-modified
problemsproblems– Tightness of modificationsTightness of modifications
Systematic exploration of IPsSystematic exploration of IPs Other domainsOther domains
– Quadratic programsQuadratic programs– Constraint programmingConstraint programming– Explore modifications of every facet of problem Explore modifications of every facet of problem
definitiondefinition
IJCAI-05 TutorialIJCAI-05 Tutorial Bounding TechniquesBounding Techniques 153153
Take-home messageTake-home message
The use of bounds is a powerful technique The use of bounds is a powerful technique for solving difficult problemsfor solving difficult problems
It has been shown that a It has been shown that a systematicsystematic approach to finding lower bounds can approach to finding lower bounds can produce useful heuristics that are not easily produce useful heuristics that are not easily discovereddiscovered
Systematically using the 2-step limit crossing Systematically using the 2-step limit crossing procedure coupled with a clear vision of all procedure coupled with a clear vision of all potential problem modifications can yield potential problem modifications can yield novel search strategies that exploit boundsnovel search strategies that exploit bounds