The Gurobi Optimizer - NUOR · E.g., at the root node Initialize pseudo-costs [Linderoth &...

The Gurobi Optimizer

Bob Bixby

Outline

Gurobi Introduction◦ Company◦ Products◦ Benchmarks

Gurobi Technology◦ Rethinking MIP◦ MIP as a bag of tricks

2© 2010 Gurobi Optimization8-Jul-11

Incorporated July, 2008 Founders: Zonghao Gu, Ed Rothberg, Bob

Bixby Product Releases:◦ Version 1.0: May 2009 Performance roughly equal to CPLEX 11.0◦ Version 2.0: October 2009◦ Version 3.0: April 2010◦ Version 4.0: November 2010◦ Version 4.5: April 2011

Gurobi Optimization

3

The Gurobi Products Gurobi Solvers◦ Mixed-Integer Programming (MIP and MIQP) Deterministic, parallel

◦ Linear and Quadratic Programming Dual and primal simplex Parallel Barrier

APIs◦ Simple command-line interface◦ Python interactive interface◦ C, C++, Java, .NET, Python callable libraries◦ All standard modeling languages

Commercial and Academic Licenses◦ Licensing options and pricing available at

www.gurobi.com

4

Company Business Model

Focus on math programming solvers Our business is helping our customers succeed with

math programming solutions Provide the best possible support from people who

know math programming Be a flexible partner Licensing models that meet user needs Clear, upfront pricing Be the technology leader Build the best available math programming

technology

5

Expanding the Reach of Optimization

6

Expanding the Reach of Optimization Multi-core Parallelism at no extra charge◦ Gurobi was the first

Free Trial Licenses◦ Automated download and license generation◦ 500 variable by 500 constraint size limit

Academic Program◦ Free single-user licenses for academics Automated download and license generation

Pay-as-you-go Licenses◦ Amazon EC2◦ Pay-by-the-day licenses

7

Academic Licenses

Over 8400 free academic licenses issued Over 60% outside the US Over 50% outside the OR community

8

MIP PerformanceBenchmarks

9

Version-to-version improvements:(Geometric mean runtime over ~800 models in our internal model set that take more than 100s to solve)◦ Gurobi 1.0 -> 2.0: 2.2X◦ Gurobi 2.0 -> 3.0: 2.9X (6.4X)◦ Gurobi 3.0 -> 4.0: 1.3X (8.3X)◦ Gurobi 4.0 -> 4.5: 1.8X (14.9X)

Continued improvement in our ability to solve difficult MIP models

MIP Performance –Gurobi Internal Test Set

10

MIP/MIQP Performance –Mittelmann Benchmarks

11

Gurobi 4.5 vs. Competition: Solve times(> 1.0 means Gurobi faster)

BenchmarkCPLEX XPRESS

P=1 P=4 P=12 P=1 P=4 P=12MIPLIB 2010 1.42X 1.19X 1.19X 1.01X 1.21X 1.16XFeasibility 3.57X - - 8.16X - -Infeasibility - - 2.62X - - 3.10XPathological - - 0.90X - - 1.04XMIQP - 2.43X - - 2.22X -

Gurobi Technology

13

Basic MIP Technology:A Fresh Look

14

Start from a clean slate Things have changed:◦ Two particular examples: “Sub-MIP” as a pervasive approach Ubiquitous parallel processing

Building From the Ground Up

15

Key recent insight for heuristics:◦ Can use MIP solver recursively◦ Solve a related model: Hopefully smaller and simpler◦ Examples: Local cuts [Applegate, Bixby, Chvatal & Cook, 2001] Local branching [Fischetti & Lodi, 2003] RINS [Danna, Rothberg, Le Pape, 2005] Solution polishing [Rothberg, 2007]

Sub-MIP As A New Paradigm

16

Rethinking MIPTree Search

17

Each node in branch-and-bound is a new MIP

Branch-and-Bound

Original model, plus several variable fixings

Can view search tree as a tree-of-trees

The nature of a sub-MIP can change dramatically

18

Gurobi MIP search tree manager built to handle multiple related trees◦ Can transform any node into the root node of a

new tree Maintains a pool of nodes from all trees◦ No need to dedicate the search to a single sub-

tree

Tree-of-Trees

19

Each tree has its own relaxation and its own strategies...◦ Presolved model for each subtree◦ Cuts specific to that subtree◦ Pseudo-costs for that subtree only◦ Symmetry detection on that submodel◦ Etc.

Captures structure that is often not visible in the original model

Tree-of-Trees

Heuristics

20

Summary of Heuristics

5 heuristics prior to solving root LP◦ 5 different variable orders, fix variables in this order

15 heuristics within tree (9 primary, several variations)◦ RINS, RINS diving, rounding, fix and dive (LP), fix and

dive (Presolve), fix and dive (simple), Lagrangian approach, pseudo costs, Hail Mary (set objective to 0)

3 solution improvement heuristics◦ Applied whenever a new integer feasible is found

Key tool:◦ Bound strengthening

21

MIP is a bag of tricks!

22

Improving a MIP Solver

Improvements can be plotted on two axes:

Generality

SpeedupBig speedups onlots of models

Modest speedupson lots of models

Big speedups ona few models

23

New Ideas

Generality

Speedup

Experiencegenerally keeps you away from here

Nine out of tenideas end up here

24

Gurobi 2.0

Generality

Dual Simplex improvements

Zero-half cuts

SOS improvements

Node files

Parallel improvements

Speedup

25

Gurobi 3.0

Generality

Speedup

BarrierNetwork cuts

Symmetry

Cut tuning

SubMIP cutsNew presolve red.

26

Disjoint Subtrees

Disjoint Subtrees

Basic principle of branching:◦ Feasible regions for child nodes after a branch should

be disjoint Not always the case Simple example – integer complementarity: x ≤ 10 b y ≤ 10 (1-b) x, y non-negative ints, x ≤ 10, y ≤ 10, b binary Branch on b: x=y=0 feasible in both children

28

Removing Overlap

Simplest way to remove overlap:◦ Modify variable bound in one subtree b=0 child: x = 0, 10 ≥ y ≥ 0 b=1 child: y = 0, 10 ≥ x ≥ 1

29

Overlap present in several models◦ 35 out of 510 models in our test set

Performance impact can be huge◦ Model neos859080 goes from 10000+ seconds

to 0.01s Median improvement for affected models

is ~1%

Performance Impact

30

A Presolve Reduction

A Presolve Reduction

Consider◦ a x + b y = c◦ x, y are integer variables◦ a, b and c are integers, a > 1◦ Assume gcd(a,b) = 1 Otherwise a Euclidean reduction is possible

◦ Observation: Then x(mod b) and y(mod a) are constants. Reduction◦ Substitute y = a z + d (d is easy to compute).◦ z has a smaller search space than y

General application◦ Can easily be extended to general “all integer”

constraints.

32

Pseudo-Cost Adjustment

33

What’s a good branching variable?◦ Superb: fractional variable infeasible in both

branch directions◦ Great: infeasible in one direction◦ Good: both directions move the objective

Expensive to predict which branches lead to infeasibility or big objective moves◦ Strong branching Truncated LP solve for every possible branch at

every node Rarely cost effective◦ Need a quick estimate

Pseudo-Costs

34

Use historical data to predict impact of a branch:◦ Record costx = obj / x for each branch Need a scheme for infeasible branches too◦ Store results in a pseudo-cost table Two entries per integer variable Average (or max) down cost Average (or max) up cost

◦ Use table to predict cost of a future branch

Pseudo-Costs

35

What do you do when there is no history?◦ E.g., at the root node

Initialize pseudo-costs [Linderoth & Savelsbergh, 1999]◦ Always compute up/down cost (using strong

branching) for new fractional variables Initialize pseudo-costs for every fractional

variable at root Reliability branching [Achterberg, Koch, &

Martin, 2002]◦ Don’t rely on historical data until pseudo-cost

for a variable has been recomputed r times

Pseudo-Cost Initialization

36

Gurobi 4.5 adjusts pseudo-costs in several ways◦ Implied pseudo-cost bounds◦ Ancestor adjustment

Pseudo-Cost Adjustment

37

Consider constraint:◦ ∑ xi = 1

Computed objective bounds:◦ x1=1 -> obj ≥ 100◦ x2=0 -> obj ≥ 110

Stronger bound for x1=1:◦ x1=1 -> x2=0 -> obj ≥ 110

In general:◦ If x=a -> y=b, then objx=a ≥ objy=b

Often violated:◦ Strong branching uses an iteration limit

Implied Pseudo-Cost Bounds

38

Consider ∑ xi = 1 again Computed objective bounds:◦ x1=0 -> obj ≥ 100◦ xj=1 -> obj ≥ 110 for all j != 1

Stronger bound for x1=0:◦ x1=0 -> xj=1 for some j != 1 -> obj ≥ 110

In general:◦ If xi=a -> xj=b for some j != i◦ Then objxi=a ≥ min(objxj=b)

Implied Pseudo-Cost Bounds

39

40

Objective move isn’t entirely the result of most recent branch variable

Ancestor Adjustment

Depends on ancestors as well

Particularly true for infeasible nodes

Adjust pseudo-costs for ancestor branches

41

Need an adjustment strategy

Ancestor Adjustment

Empirically, adjustment should decrease as you move up the tree

Our approach: Exponential

backoff ½ for parent, ¼ for

grandparent, etc.

Product Roadmap

Projected Gurobi Roadmap

Version 4.X – November 2011◦ LP & MIP performance release

Version 5.0 – April 2012◦ SOCP and MISOCP◦ MIP performance enhancements

Version 6.0◦ New features under discussion◦ MIP performance enhancements

43

Thank You