A linear-time algorithm for the generalized TRS basedon a convex quadratic reformulation

Alex L. Wang Fatma Kılınc-Karzan

Carnegie Mellon University, PA, USA

ICCOPT 2019

Alex L. Wang, Fatma Kılınc-Karzan Linear-time alg. for the GTRS 1 / 19

1 Introduction

2 Convex hull result

3 Convex quadratic reformulation of the GTRS and algorithms

Alex L. Wang, Fatma Kılınc-Karzan Linear-time alg. for the GTRS 2 / 19

The Generalized Trust Region Subproblem (GTRS)

• qobj, qcons : Rn → R are nonconvex quadratic functions

Opt := infx∈Rn{qobj(x) | qcons(x) ≤ 0}

• qobj(x) = x>Aobjx + 2b>objx + cobj

• qcons(x) = x>Aconsx + 2b>consx + ccons

Alex L. Wang, Fatma Kılınc-Karzan Linear-time alg. for the GTRS 3 / 19

Motivation

• Applications• Nonconvex quadratic integer programs, signal processing, compressed

sensing, robust optimization, trust-region methods

• Surprisingly simple/beautiful theory• Semidefinite programming (SDP) relaxation is tight [FY79] =⇒

polynomial-time algorithm• Connections between GTRS and generalized eigenvalues• Special instance of quadratically-constrained quadratic

programming (QCQP)

Alex L. Wang, Fatma Kılınc-Karzan Linear-time alg. for the GTRS 4 / 19

Related work

• Convex reformulations of the GTRS in lifted spaces[BT96; BH14]

• Algorithms for the GTRS assuming exact eigen-procedures[PW14; FST18; JLW18; JL19; AN19]

• A linear-time algorithm for the GTRS[JL18]

• Convex hull results[Yıl09; MV17]

Alex L. Wang, Fatma Kılınc-Karzan Linear-time alg. for the GTRS 5 / 19

Results/Outline

Under “mild assumptions”

• Convex hull result =⇒ convex quadratic reformulation [JL19]

• New linear-time algorithm for approximating the GTRS

• Results extend to equality-, interval-, and hollow-constrained variantsof the GTRS

Alex L. Wang, Fatma Kılınc-Karzan Linear-time alg. for the GTRS 6 / 19

1 Introduction

2 Convex hull result

3 Convex quadratic reformulation of the GTRS and algorithms

Alex L. Wang, Fatma Kılınc-Karzan Linear-time alg. for the GTRS 7 / 19

Suffices to optimize over convex hull of epigraph

infx∈Rn{qobj(x) | qcons(x) ≤ 0}

= inf(x ,t)∈Rn+1

{t

∣∣∣∣ qobj(x) ≤ tqcons(x) ≤ 0

}=: inf

x ,t{t | (x , t) ∈ S}

= infx ,t{t | (x , t) ∈ conv(S)}

Alex L. Wang, Fatma Kılınc-Karzan Linear-time alg. for the GTRS 8 / 19

A pencil of quadratics

• Main object of analysis

q(γ, x) := qobj(x) + γqcons(x)

• A(γ) := Aobj + γAcons

• S(γ) := {(x , t) | q(γ, x) ≤ t}

Exercise

S =⋂γ≥0 S(γ)

{(x , t)

∣∣∣∣ qobj(x) ≤ tqcons(x) ≤ 0

}?=⊆⊇=

⋂γ≥0

{(x , t) | qobj(x) + γqcons(x) ≤ t}

Alex L. Wang, Fatma Kılınc-Karzan Linear-time alg. for the GTRS 9 / 19

A pencil of quadratics

γ0

• S(γ) := {(x , t) | q(γ, x) ≤ t}• S(0) = {(x , t) | qobj(x) ≤ t}• S(large number) ≈ {(x , t) | qcons(x) ≤ 0}• S ≈ S(0) ∩ S(large number)

≈ ∩

Alex L. Wang, Fatma Kılınc-Karzan Linear-time alg. for the GTRS 10 / 19

Convex hull result

γ0 γ− γ+

• Define Γ := {γ ≥ 0 | S(γ) is convex} = {γ ≥ 0 |A(γ) � 0}• Define [γ−, γ+] := Γ

• conv(S) = S(γ−) ∩ S(γ+)

= ∩

Alex L. Wang, Fatma Kılınc-Karzan Linear-time alg. for the GTRS 11 / 19

Convex hull result

Theorem

Suppose qobj and qcons are both nonconvex and Γ is nonempty, i.e., thereexists γ ≥ 0 such that A(γ) � 0. Then Γ can be written Γ = [γ−, γ+] and

conv(S) = S(γ−) ∩ S(γ+) =

{(x , t)

∣∣∣∣ q(γ−, x) ≤ tq(γ+, x) ≤ t

}

= ∩

Alex L. Wang, Fatma Kılınc-Karzan Linear-time alg. for the GTRS 12 / 19

1 Introduction

2 Convex hull result

3 Convex quadratic reformulation of the GTRS and algorithms

Alex L. Wang, Fatma Kılınc-Karzan Linear-time alg. for the GTRS 13 / 19

Convex quadratic reformulation

• We can reformulate

Opt = inf(x ,t){t | (x , t) ∈ conv(S)}

= inf(x ,t)

{t

∣∣∣∣ q(γ−, x) ≤ tq(γ+, x) ≤ t

}= inf

xmax {q(γ−, x), q(γ+, x)}

• Algorithmic challenges• γ− and γ+ are not given• Only have approximate (generalized) eigen-procedures• Smooth minimax framework [Nes18] requires efficiently computing a

prox mapping• How to recover a solution to the GTRS?

Alex L. Wang, Fatma Kılınc-Karzan Linear-time alg. for the GTRS 14 / 19

A linear-time algorithm for the GTRS

• Algorithm idea (assume A0 and A1 are “well-conditioned”)• N is the number of nonzero entries in A0 and A1

• p is the failure probability• Approximate γ− and γ+ to “high enough accuracy”

O

(N√ε

log

(n

p

)log

(1

ε

))• Approximately solve a smooth minimax problem

O

(N√ε

)

Alex L. Wang, Fatma Kılınc-Karzan Linear-time alg. for the GTRS 15 / 19

A linear-time algorithm for the GTRS

Theorem

There exists an algorithm, which given nonconvex quadratics qobj andqcons satisfying

• there exists γ ≥ 0 such that A(γ) � 0 and

• “mild” regularity assumptions,

outputs an ε-approximate optimizer to the GTRS with probability ≥ 1− p.This algorithm runs in time

≈ O

(N√ε

log

(n

p

)log

(1

ε

))where N is the number of nonzero entries in Aobj and Acons.

Alex L. Wang, Fatma Kılınc-Karzan Linear-time alg. for the GTRS 16 / 19

Recap

• Want to optimize the GTRS: infx {qobj(x) | qcons(x) ≤ 0}• Studied a pencil of quadratics q(γ, x)

• Gave an explicit description of conv(S)

• Convex quadratic reformulation!

• Gave a linear (in N) time algorithm for solving the GTRS and itsvariants

Alex L. Wang, Fatma Kılınc-Karzan Linear-time alg. for the GTRS 17 / 19

Future directions

• Can we generalize techniques in this paper to handle more than onequadratic constraints?

• What are the “right” regularity parameters?

Alex L. Wang, Fatma Kılınc-Karzan Linear-time alg. for the GTRS 18 / 19

Thank you. Questions?

Slides and Mathematica notebookcs.cmu.edu/~alw1/iccopt.html

PreprintAlex L. Wang and Fatma Kılınc-Karzan. The Generalized Trust RegionSubproblem: solution complexity and convex hull results. Tech. rep.arXiv:1907.08843. ArXiV, 2019. url: arxiv.org/abs/1907.08843

Alex L. Wang, Fatma Kılınc-Karzan Linear-time alg. for the GTRS 19 / 19

References I

Satoru Adachi and Yuji Nakatsukasa. “Eigenvalue-basedalgorithm and analysis for nonconvex QCQP with oneconstraint”. In: Mathematical Programming 173.1 (2019),pp. 79–116.

Aharon Ben-Tal and Dick den Hertog. “Hidden conic quadraticrepresentation of some nonconvex quadratic optimizationproblems”. In: Mathematical Programming 143.1 (2014),pp. 1–29.

Aharon Ben-Tal and Marc Teboulle. “Hidden convexity in somenonconvex quadratically constrained quadratic programming”.In: Mathematical Programming 72.1 (1996), pp. 51–63.

S. Fallahi, M. Salahi, and T. Terlaky. “Minimizing an indefinitequadratic function subject to a single indefinite quadraticconstraint”. In: Optimization 67.1 (2018), pp. 55–65.

Alex L. Wang, Fatma Kılınc-Karzan Linear-time alg. for the GTRS 19 / 19

References II

Alexander L. Fradkov and Vladimir A. Yakubovich. “TheS-procedure and duality relations in nonconvex problems ofquadratic programming”. In: Vestn. LGU, Ser. Mat., Mekh.,Astron 6.1 (1979), pp. 101–109.

Rujun Jiang and Duan Li. A linear-time algorithm forgeneralized trust region problems. Tech. rep. arXiv:1807.07563.ArXiV, 2018. url: https://arxiv.org/abs/1807.07563.

Rujun Jiang and Duan Li. “Novel reformulations and efficientalgorithms for the Generalized Trust Region Subproblem”. In:SIAM Journal on Optimization 29.2 (2019), pp. 1603–1633.

Rujun Jiang, Duan Li, and Baiyi Wu. “SOCP reformulation forthe Generalized Trust Region Subproblem via a canonical formof two symmetric matrices”. In: Mathematical Programming169.2 (2018), pp. 531–563.

Alex L. Wang, Fatma Kılınc-Karzan Linear-time alg. for the GTRS 19 / 19

References III

Sina Modaresi and Juan Pablo Vielma. “Convex hull of twoquadratic or a conic quadratic and a quadratic inequality”. In:Mathematical Programming 164.1-2 (2017), pp. 383–409.

Yurii Nesterov. Lectures on convex optimization (2nd Ed.)Springer Optimization and Its Applications. Basel, Switzerland:Springer International Publishing, 2018.

Ting Kei Pong and Henry Wolkowicz. “The Generalized TrustRegion Subproblem”. In: Computational Optimization andApplications 58.2 (2014), pp. 273–322.

Ugur Yıldıran. “Convex hull of two quadratic constraints is anLMI set”. In: IMA Journal of Mathematical Control andInformation 26.4 (2009), pp. 417–450.

Alex L. Wang, Fatma Kılınc-Karzan Linear-time alg. for the GTRS 19 / 19