A Tighter Piecewise McCormick

Relaxation for Bilinear Problems

Pedro M. CastroCentro de Investigação Operacional

Faculdade de Ciências

Universidade de Lisboa

MINLP 2014

Problem definition (NLP)

June 3, 2014 Tighter Piecewise McCormick Relaxation 2

𝑎𝑖𝑗𝑞- parameters

𝒙 – vector of continuous variables𝐵𝐿𝑞 - (𝑖, 𝑗) index set

𝑖 ≠ 𝑗 – strictly bilinear problems𝑖 = 𝑗 – can be allowed (quadratic problems)ℎ𝑞(𝑥)- linear function in 𝑥

Relaxation provides a lower bound

min 𝑓0(𝑥)

𝑓𝑞(𝑥) =

(𝑖,𝑗)∈𝐵𝐿𝑞

𝑎𝑖𝑗𝑞𝑥𝑖𝑥𝑗 + ℎ𝑞(𝑥) 𝑞 ∈ 𝑄

• Bilinear program

0 ≤ 𝑥𝐿 ≤ 𝑥 ≤ 𝑥𝑈

𝑥 ∈ ℝ𝑚

𝑓𝑞(𝑥) ≤ 0 𝑞 ∈ 𝑄\{0}

(P)

MINLP 2014

Introduction

• Bilinear problems occur in a variety of applications– Process network problems (Quesada & Grossmann, 1995)

– Water networks (Bagajewicz, 2000)

– Pooling and blending (Haverly, 1978)

• Non-convex, leading to multiple local solutions– Gradient based solvers unable to certify optimality

• Need for global optimization algorithms– What do they have in common?

• Linear (LP) or mixed-integer linear (MILP) relaxation of (P) → LB– LP: Standard McCormick envelopes (1976)

– MILP: Piecewise McCormick envelopes (Bergamini et al. 2005)

– MILP: Multiparametric disaggregation (Teles et al. 2013; Kolodziej et al. 2013)

• Solution of (P) with fast local solver → UB– Using LB as starting point

• Tight relaxation critical to ensure convergence– Relative optimality gap (UB-LB)/LB<

June 3, 2014 3Tighter Piecewise McCormick Relaxation

MINLP 2014

Generalized Disjunctive Program (Balas, 1979; Raman & Grossmann, 1994)

Piecewise McCormick relaxation

• Domain of divided into 𝑁 uniform partitions

June 3, 2014 4Tighter Piecewise McCormick Relaxation

𝑤𝑖𝑗 = 𝑥𝑖𝑥𝑗

Optimization will

pick single partition. . .

𝑥𝑗𝐿

𝑥𝑗𝑈

𝑥𝑗,1𝐿 𝑥𝑗,2

𝐿 𝑥𝑗,𝑁𝐿

𝑥𝑗,𝑁𝑈𝑥𝑗,𝑁−1

𝑈𝑥𝑗,2𝑈𝑥𝑗,1

𝑈𝑛 = 1 𝑛 = 2 𝑛 = 𝑁

𝑦𝑗,1 = 𝑡𝑟𝑢𝑒 𝑦𝑗,2 = 𝑡𝑟𝑢𝑒 𝑦𝑗,𝑁 = 𝑡𝑟𝑢𝑒∨ ∨ ∨. . .

𝑥𝑗𝑈𝑥𝑗

𝐿 𝑥𝑗

𝑥𝑖

Convex

hull

1.25

1.45

1.65

1.85

2.05

2.25

2.45

200 250 300 350 400

𝑁 = 2

1.25

1.45

1.65

1.85

2.05

2.25

2.45

200 250 300 350 400

𝑁 = 20

Standard McCormick𝑁 = 1

∨𝑛

𝑦𝑗𝑛

𝑤𝑖𝑗 ≥ 𝑥𝑖 ⋅ 𝑥𝑗𝑛𝐿 + 𝑥𝑖

𝐿 ⋅ 𝑥𝑗 − 𝑥𝑖𝐿 ⋅ 𝑥𝑗𝑛

𝐿

𝑤𝑖𝑗 ≥ 𝑥𝑖 ⋅ 𝑥𝑗𝑛𝑈 + 𝑥𝑖

𝑈 ⋅ 𝑥𝑗 − 𝑥𝑖𝑈 ⋅ 𝑥𝑗𝑛

𝑈

𝑤𝑖𝑗 ≤ 𝑥𝑖 ⋅ 𝑥𝑗𝑛𝐿 + 𝑥𝑖

𝑈 ⋅ 𝑥𝑗 − 𝑥𝑖𝑈 ⋅ 𝑥𝑗𝑛

𝐿

𝑤𝑖𝑗 ≤ 𝑥𝑖 ⋅ 𝑥𝑗𝑛𝑈 + 𝑥𝑖

𝐿 ⋅ 𝑥𝑗 − 𝑥𝑖𝐿 ⋅ 𝑥𝑗𝑛

𝑈

𝑥𝑗𝑛𝐿 ≤ 𝑥𝑗 ≤ 𝑥𝑗𝑛

𝑈

𝑥𝑖𝐿 ≤ 𝑥𝑖 ≤ 𝑥𝑖

𝑈

𝑥𝑗𝑛𝐿 = 𝑥𝑗

𝐿 + (𝑥𝑗𝑈 − 𝑥𝑗

𝐿) ⋅ (𝑛 − 1)/𝑁

𝑥𝑗𝑛𝑈 = 𝑥𝑗

𝐿 + (𝑥𝑗𝑈 − 𝑥𝑗

𝐿) ⋅ 𝑛/𝑁

(PR-MILP)

MINLP 2014

New tighter piecewise relaxation

• Basic idea– Use partition-dependent

bounds also for variable 𝑥𝑖• 𝑥𝑖𝑗𝑛

𝐿 /𝑥𝑖𝑗𝑛𝑈 lower/upper bound

when 𝑥𝑗 constrained to 𝑛

– Better bounds tighter relaxation lower gap

June 3, 2014 5Tighter Piecewise McCormick Relaxation

∨𝑛

𝑦𝑗𝑛

𝑤𝑖𝑗 ≥ 𝑥𝑖 ⋅ 𝑥𝑗𝑛𝐿 + 𝑥𝑖𝑗𝑛

𝐿 ⋅ 𝑥𝑗 − 𝑥𝑖𝑗𝑛𝐿 ⋅ 𝑥𝑗𝑛

𝐿

𝑤𝑖𝑗 ≥ 𝑥𝑖 ⋅ 𝑥𝑗𝑛𝑈 + 𝑥𝑖𝑗𝑛

𝑈 ⋅ 𝑥𝑗 − 𝑥𝑖𝑗𝑛𝑈 ⋅ 𝑥𝑗𝑛

𝑈

𝑤𝑖𝑗 ≤ 𝑥𝑖 ⋅ 𝑥𝑗𝑛𝐿 + 𝑥𝑖𝑗𝑛

𝑈 ⋅ 𝑥𝑗 − 𝑥𝑖𝑗𝑛𝑈 ⋅ 𝑥𝑗𝑛

𝐿

𝑤𝑖𝑗 ≤ 𝑥𝑖 ⋅ 𝑥𝑗𝑛𝑈 + 𝑥𝑖𝑗𝑛

𝐿 ⋅ 𝑥𝑗 − 𝑥𝑖𝑗𝑛𝐿 ⋅ 𝑥𝑗𝑛

𝑈

𝑥𝑖𝑗𝑛𝐿 ≤ 𝑥𝑖 ≤ 𝑥𝑖𝑗𝑛

𝑈

𝑥𝑗𝑛𝐿 ≤ 𝑥𝑗 ≤ 𝑥𝑗𝑛

𝑈

𝑥𝑗𝑛𝐿 = 𝑥𝑗

𝐿 + (𝑥𝑗𝑈 − 𝑥𝑗

𝐿) ⋅ (𝑛 − 1)/𝑁

𝑥𝑗𝑛𝑈 = 𝑥𝑗

𝐿 + (𝑥𝑗𝑈 − 𝑥𝑗

𝐿) ⋅ 𝑛/𝑁

(PRT-MILP)

Convex

hull

min 𝑧𝑅 = 𝑓0 𝑥 =

𝑖,𝑗 ∈𝐵𝐿

𝑎𝑖𝑗0𝑤𝑖𝑗 + ℎ0 𝑥

𝑓𝑞 𝑥 =

𝑖,𝑗 ∈𝐵𝐿

𝑎𝑖𝑗𝑞𝑤𝑖𝑗 + ℎ𝑞 𝑥 ≤ 0 ∀𝑞 ∈ 𝑄\{0}

𝑥𝑗 = 𝑛 𝑥𝑗𝑛 𝑛 𝑦𝑗𝑛 = 1

∀ {𝑗|(𝑖, 𝑗) ∈ 𝐵𝐿}

𝑥𝑗𝑛𝐿 = 𝑥𝑗

𝐿 +𝑥𝑗𝑈 − 𝑥𝑗

𝐿 ∙ 𝑛 − 1

𝑁

𝑥𝑗𝑛𝑈 = 𝑥𝑗

𝐿 +𝑥𝑗𝑈 − 𝑥𝑗

𝐿 ∙ 𝑛

𝑁𝑥𝑗𝑛𝐿 ⋅ 𝑦𝑗𝑛 ≤ 𝑥𝑗𝑛 ≤ 𝑥𝑗𝑛

𝑈 ⋅ 𝑦𝑗𝑛

∀ 𝑗 𝑖, 𝑗 ∈ 𝐵𝐿 , 𝑛

𝑤𝑖𝑗 ≥ 𝑛

𝑥𝑖𝑗𝑛 ⋅ 𝑥𝑗𝑛𝐿 + 𝑥𝑖𝑗𝑛

𝐿 ⋅ 𝑥𝑗𝑛 − 𝑥𝑖𝑗𝑛𝐿 ⋅ 𝑥𝑗𝑛

𝐿 ⋅ 𝑦𝑗𝑛

𝑤𝑖𝑗 ≥ 𝑛

𝑥𝑖𝑗𝑛 ⋅ 𝑥𝑗𝑛𝑈 + 𝑥𝑖𝑗𝑛

𝑈 ⋅ 𝑥𝑗𝑛 − 𝑥𝑖𝑗𝑛𝑈 ⋅ 𝑥𝑗𝑛

𝑈 ⋅ 𝑦𝑗𝑛

𝑤𝑖𝑗 ≤ 𝑛

𝑥𝑖𝑗𝑛 ⋅ 𝑥𝑗𝑛𝐿 + 𝑥𝑖𝑗𝑛

𝑈 ⋅ 𝑥𝑗𝑛 − 𝑥𝑖𝑗𝑛𝑈 ⋅ 𝑥𝑗𝑛

𝐿 ⋅ 𝑦𝑗𝑛

𝑤𝑖𝑗 ≤ 𝑛

𝑥𝑖𝑗𝑛 ⋅ 𝑥𝑗𝑛𝑈 + 𝑥𝑖𝑗𝑛

𝐿 ⋅ 𝑥𝑗𝑛 − 𝑥𝑖𝑗𝑛𝐿 ⋅ 𝑥𝑗𝑛

𝑈 ⋅ 𝑦𝑗𝑛

𝑥𝑖 = 𝑛 𝑥𝑖𝑗𝑛

∀ (𝑖, 𝑗)

𝑥𝑖𝑗𝑛𝐿 ⋅ 𝑦𝑗𝑛 ≤ 𝑥𝑖𝑗𝑛 ≤ 𝑥𝑖𝑗𝑛

𝑈 ⋅ 𝑦𝑗𝑛 ∀ (𝑖, 𝑗) ∈ 𝐵𝐿, 𝑛 ∈ {1, … , 𝑁}

MINLP 2014

How to generate bounds 𝑥𝑖𝑗𝑛𝐿 & 𝑥𝑖𝑗𝑛

𝑈 ?

• Optimality bound contraction with McCormick envelopes– Solve multiple instances of LP problem (BC), two per:

• Non-partitioned variable 𝑥𝑖• Partitioned variable 𝑥𝑗• Partition 𝑛

– Focus on regions that can actually improve current UB• 𝑧´ obtained from solving (P) with local solver

– (BC) infeasible? Remove partition 𝑛∗ of 𝑗∗ from (PRT-MILP)

– Computation can be time consuming

June 3, 2014 6Tighter Piecewise McCormick Relaxation

𝑤𝑖𝑗 ≥ 𝑥𝑖 ⋅ 𝑥𝑗𝐿 + 𝑥𝑖

𝐿 ⋅ 𝑥𝑗 − 𝑥𝑖𝐿 ⋅ 𝑥𝑗

𝐿

𝑤𝑖𝑗 ≥ 𝑥𝑖 ⋅ 𝑥𝑗𝑈 + 𝑥𝑖

𝑈 ⋅ 𝑥𝑗 − 𝑥𝑖𝑈 ⋅ 𝑥𝑗

𝑈

𝑤𝑖𝑗 ≤ 𝑥𝑖 ⋅ 𝑥𝑗𝐿 + 𝑥𝑖

𝑈 ⋅ 𝑥𝑗 − 𝑥𝑖𝑈 ⋅ 𝑥𝑗

𝐿

𝑤𝑖𝑗 ≤ 𝑥𝑖 ⋅ 𝑥𝑗𝑈 + 𝑥𝑖

𝐿 ⋅ 𝑥𝑗 − 𝑥𝑖𝐿 ⋅ 𝑥𝑗

𝑈

∀ (𝑖, 𝑗) ∈ 𝐵𝐿

𝑥𝑗∗𝐿 = 𝑥𝑗∗𝑛∗

𝐿 ≤ 𝑥𝑗∗ ≤ 𝑥𝑗∗𝑛∗𝑈 = 𝑥𝑗∗

𝑈

𝑥𝐿 ≤ 𝑥 ≤ 𝑥𝑈

𝑓𝑞(𝑥) =

𝑖,𝑗 ∈𝐵𝐿𝑞

𝑎𝑖𝑗𝑞𝑤𝑖𝑗 + ℎ𝑞(𝑥) ≤ 0 ∀𝑞 ∈ 𝑄\{0}

𝑥𝑖∗𝑗∗𝑛∗𝐿 ∶= min 𝑥𝑖∗ (𝑥𝑖∗𝑗∗𝑛∗

𝑈 ∶= max 𝑥𝑖∗

(BC)

𝑓0 𝑥 =

𝑖,𝑗 ∈𝐵𝐿0

𝑎𝑖𝑗0𝑤𝑖𝑗 + ℎ0(𝑥) ≤ 𝑧´

MINLP 2014

New optimization algorithm

• Key features

– User selects:

• 𝑁 partitions

• Partitioned

variables 𝑥𝑗

– Preliminary bound

contraction stage

– Bounds updated at

different levels

– LB from

MILP relaxation

– UB from NLP (single

starting point)

– Computes an

optimality gap

June 3, 2014 Tighter Piecewise McCormick Relaxation 7

Given:

partitions

, , , , ,

SOLVE (P)with local solver

upper bound

SOLVE (standard bound contract)

update, , ,

Initialization

SOLVE (BC)

Feasible?remove partitionNOtighter bounds

,

YES

YESupdate bounds

NO

YESupdate bounds

NO

NO

SOLVE(PRT-MILP)

YES

OutputOptimal solution

Optimality gap

Time Elapsed

lower bound

SOLVE (P)with local solver

upper bound

Given:

partitions

, , , , ,

SOLVE (P)with local solver

upper bound

SOLVE (standard bound contract)

update, , ,

Initialization

SOLVE (BC)

Feasible?remove partitionNOtighter bounds

,

YES

YESupdate bounds

NO

YESupdate bounds

NO

NO

SOLVE(PRT-MILP)

YES

OutputOptimal solution

Optimality gap

Time Elapsed

lower bound

SOLVE (P)with local solver

upper bound

MINLP 2014

Illustrative example P1

June 3, 2014 Tighter Piecewise McCormick Relaxation 8

min 𝑓0 𝑥 = −𝑥1 + 𝑥1𝑥2 − 𝑥2

−6𝑥1 + 8𝑥2 ≤ 33𝑥1 − 𝑥2 ≤ 3

0 ≤ 𝑥1, 𝑥2 ≤ 1.5 Standard optimality bound contraction

0.357143 ≤ 𝑥1 ≤ 1.375

0 ≤ 𝑥2 ≤ 1.26

Partitioned variable: 𝑥1𝐵𝐿 = {(2,1)}

Non-partitioned variable: 𝑥2

Partition 𝑛 𝑥1𝒏𝐿 𝑥1𝒏

𝑼 𝑥21𝒏𝐿 𝑥21𝒏

𝑼

1 0.357143 0.696429 LP is infeasible

2 0.696429 1.035714 0 1.137609

3 1.035714 1.375 0.107143 1.195150

Partition-dependent bound contraction 𝑁 = 3

𝑧 = 𝑓0 𝑥 = −1.083333

Global optimum

𝑧𝑅 = −1.185207

(PR-MILP) relaxation

𝑧𝑅 = −1.169619

(PRT-MILP) relaxation

Gap

8.60%

7.38%

MINLP 2014

How about using more partitions?

• Up to two orders of magnitude reduction in optimality gap– Major increase in total computational time

– Still, new approach performs best!

June 3, 2014 9Tighter Piecewise McCormick Relaxation

Relaxation Partitions (𝑁) 15 150 1500 7500

(PR-MILP)Total CPUs 0.93 1.60 3.63 302

Optimality gap 1.67% 0.188% 0.0189% 0.0038%

(PRT-MILP)Optimality gap 0.77% 0.010% 0.0002%

Total CPUs 3.63 27.8 267

P1

Relaxation Partitions (𝑁) 9 90 900 2700

(PR-MILP)Total CPUs 0.83 1.2 7.76 478

Optimality gap 8.41% 0.84% 0.0830% 0.0278%

(PRT-MILP)Optimality gap 0.84% 0.02% 0.0002%

Total CPUs 3.33 24.0 223

P2

min 6𝑥12 + 4𝑥2

2 − 2.5𝑥1𝑥2

𝑥1𝑥2 − 8 ≥ 0

1 ≤ 𝑥1, 𝑥2 ≤ 10

MINLP 2014

Results for larger test problems

• 16 easiest water-using design problems– New (PRT-MILP) approach leads to lower gaps at termination (3600 CPUs)

– Faster in 56% of the cases (finest partition level)

– Single failure when finding best-known solution for (P)

June 3, 2014 10Tighter Piecewise McCormick Relaxation

Number of variables (PR-MILP) (PRT-MILP)

Non-partitioned |𝐼| Partitioned |𝐽| Partitions (𝑁) Solution Total CPUs Gap Gap Total CPUs Solution

Ex2 20 1610

74.469918.8 0.1250% 0.0484% 141

74.469950 3609 0.1245% 0.0103% 1536

Ex6 30 2510

142.081644.7 0.0435% 0.0110% 205

142.081650 3614 0.1379% 0.0041% 1090

Ex8 30 2010

164.489818.8 0.7134% 0.2132% 207

164.4898100 3613 1.9508% 0.0130% 1803

Ex10 15 910

169.11736.69 0.0160% 0.0008% 45.3

169.1173100 809 0.0023% 0.0000% 323

Ex14 32 2410 329.9689 282 1.2259% 1.0499% 447

329.569820 329.9566 3615 2.3368% 0.9206% 4270

Ex15 40 3010 361.5177 231 0.7963% 0.7427% 723 361.5177

20 361.6856 3618 0.9361% 0.7779% 4486 361.6856

Ex16 24 1610

285.934317.4 1.1105% 0.8861% 212

285.934350 3610 1.5755% 0.0526% 2042S

ub

se

t o

f p

rob

lem

s s

ele

cte

d fo

r ill

ustr

atio

n

MINLP 2014

How to determine the number of partitions?

• Quality of relaxation increases with 𝑁– Careful not to generate intractable MILPs (Gounaris et al. 2009)

• Several works identify most appropriate value for a particular problem(Misener et al. 2011; Wittmann-Holhbein and Pistikopoulos, 2013; 2014)

– No method to estimate 𝑁 as a function of problem complexity

– Needed to provide a fair comparison with commercial solvers

• Proposed formula meets important criteria:– Returns an integer value

– Ensures minimum of two partitions• So as to benefit from piecewise relaxation scheme

– Reduces 𝑁 with increase in problem size• Roughly inversely proportional (r2=0.76)

June 3, 2014 11Tighter Piecewise McCormick Relaxation

𝑁 = 1 +𝛼

|𝐼| ∙ |𝐽|𝛼 = 1.8𝐸4

MINLP 2014

Comparison to commercial solvers

• Performance profiles(Dolan & Moré, 2002)

– KPI: Optimality gap

June 3, 2014 12Tighter Piecewise McCormick Relaxation

0

0.2

0.4

0.6

0.8

1

0 1 2 3 4 5 6 7 8 9 10

Cum

ula

tive d

istr

ibution f

unction

(PR-MILP)

(PRT-MILP)

GloMIQO 2.3

BARON 12.3.3

GAMS 24.1.3, CPLEX 12.5.1, GloMIQO 2.3,

BARON 12.3.3, Intel Core i7-3770 (3.07 GHz),

8 GB RAM, Windows 7 64-bit

Termination criteria: gap=0.0001%, Time=3600 CPUs

• (PRT-MILP) better (PR-MILP)

• GloMIQO outperforms BARON

# Failures finding best-known solution (34 problems)

Algorithm GloMIQO (PRT-MILP) BARON (PR-MILP)

Suboptimal 3 5 2 7

No solution 0 0 4 0

Total 3 5 6 7

MINLP 2014

Why not bivariate partitioning?

• Domain of 𝑥𝑖 and 𝑥𝑗 known a priori for each 2-D partition

– Slightly better performance than univariate partitioning for P1

– Gap for 1252 partitions one order of magnitude larger vs. (PRT-MILP)

June 3, 2014 13Tighter Piecewise McCormick Relaxation

𝑛=1

𝑁

𝑛´=1

𝑁

𝑦𝑖𝑛𝑗𝑛´

𝑤𝑖𝑗 ≥ 𝑥𝑖 ⋅ 𝑥𝑗𝑛´𝐿 + 𝑥𝑖𝑛

𝐿 ⋅ 𝑥𝑗 − 𝑥𝑖𝑛𝐿 ⋅ 𝑥𝑗𝑛´

𝐿

𝑤𝑖𝑗 ≥ 𝑥𝑖 ⋅ 𝑥𝑗𝑛´𝑈 + 𝑥𝑖𝑛

𝑈 ⋅ 𝑥𝑗 − 𝑥𝑖𝑛𝑈 ⋅ 𝑥𝑗𝑛´

𝑈

𝑤𝑖𝑗 ≤ 𝑥𝑖 ⋅ 𝑥𝑗𝑛´𝐿 + 𝑥𝑖𝑛

𝑈 ⋅ 𝑥𝑗 − 𝑥𝑖𝑛𝑈 ⋅ 𝑥𝑗𝑛´

𝐿

𝑤𝑖𝑗 ≤ 𝑥𝑖 ⋅ 𝑥𝑗𝑛´𝑈 + 𝑥𝑖𝑛

𝐿 ⋅ 𝑥𝑗 − 𝑥𝑖𝑛𝐿 ⋅ 𝑥𝑗𝑛´

𝑈

𝑥𝑖𝑛𝐿 ≤ 𝑥𝑖 ≤ 𝑥𝑖𝑛

𝑈

𝑥𝑗𝑛´𝐿 ≤ 𝑥𝑗 ≤ 𝑥𝑗𝑛´

𝑈

∀(𝑖, 𝑗)

(PR

-BV

-MIL

P)

Relaxation Partitions (𝑁) 16 144 2500 10000

Univariate

(PR-MILP)

Total CPUs 0.88 1.42 9.11 429

Optimality gap 1.70% 0.197% 0.0114% 0.0029%

𝑵 4 12 50 100 125

Bivariate

(PR-BV-MILP)

Optimality gap 1.28% 0.188% 0.0078% 0.0029% 0.0018%

Total CPUs 0.93 0.91 7.55 217 469

MINLP 2014

Conclusions

• Bilinear problems tackled through piecewise relaxation (PR)

• Novel approach with partition-dependent boundsfor all bilinear variables– Significant improvement in relaxation quality

– Need for extensive optimality-based bound contraction (BC)• Total computational time is sometimes lower

• New algorithm outperforms state-of-the-art global solvers– Specific class of water-using network design problems

• PR and BC schemes should be used to greater extent– Integrating with spatial B&B subject of future work

• See CACE paper for further details

• Acknowledgments:– Luso-American Foundation (2013 Portugal-U.S. Research Networks)

– Fundação para a Ciência e Tecnologia (Investigador FCT 2013)

June 3, 2014 14Tighter Piecewise McCormick Relaxation