Matrix

Rank

Min

imiza

tion

with

Applica

tions

Maryam

Fazel

Haith

amH

indi

Step

hen

Boyd

Inform

ationSystem

sLab

Electrical

Engin

eering

Dep

artmen

tStan

fordU

niversity

8/2001

ACC’0

1

Outlin

e

•Ran

kM

inim

izationProb

lem(R

MP)

•Exam

ples

•Solu

tionm

ethods

ACC’0

11

Rank

Min

imiza

tion

Pro

ble

m(R

MP)

min

imize

Rank

X

subject

toX∈C,

X∈

Rm×

nis

the

optim

izationvariab

le;C

iscon

vexset

•RM

Pis

diffi

cult

nonco

nve

xprob

lem(N

P-h

ard)

•RM

Parises

inm

any

application

areas

•usu

alm

eanin

g:find

simplest

orm

inim

um

order

system;m

odel

with

fewest

param

eters...(O

ccam’s

razor)

this

talk

:new

meth

ods

to(a

ppro

ximate

ly)so

lveRM

P

ACC’0

12

Maxim

um

sparsity

pro

ble

m

importan

tsp

ecialcase

ofRM

P:variab

leX

=dia

g(x

)

then

Rank

X=

Card(x

),num

ber

ofnon

zerox

i

RM

Pred

uces

tofindin

gth

esp

arsest

vectorin

convex

setC:

min

imize

Card(x

)su

bject

tox∈C

mean

ing:

find

simplest

model,

design

with

fewest

compon

ents,

sparse

signal

representation

;extract

sparse

signals

fromnoise,

...

ACC’0

13

Constra

ined

facto

ranalysis

find

min

imum

rank

covariance

matrix

closeto

measu

redΣ̂

,con

sistent

with

priorin

fom

inim

izeR

ank

Σ

subject

to‖Σ

−Σ̂‖≤

ε

Σ=

ΣT�

0Σ∈C

Σ∈

Rn×

nis

variable,

Cis

priorin

formation

,ε

istoleran

ce

Rank

Σis

num

ber

offa

ctors

that

explain

Σ

with

out

lastcon

straint,

cansolve

byeigen

value

decom

position

;but

general

caseis

diffi

cult

ACC’0

14

Min

imum

ord

erco

ntro

llerdesig

n

find

min

imum

order

controller

that

achieves

specifi

edclosed

-loop

perform

ance

min

imize

Rank

[

XI

IY

]

subject

to

[

XI

IY

]

�0

other

LM

Isin

X,Y

with

variables

X=

XT,

Y=

YT∈

Rn×

n,possib

lyoth

ers

ACC’0

15

Min

imum

ord

er

system

realiza

tion

find

min

imum

order

systemth

atsatisfi

estim

e-dom

ainsp

ecs(rise-tim

e,slew

-rate,oversh

oot,

settling-tim

e,...)

min

imize

Rank

h1

h2

···h

n

h2

h3

···h

n+

1...

......

hn

hn+

1···

h2n−

1

subject

toF

(h1 ,...,h

n)�

g

variables

areim

pulse

respon

seh

1 ,...,h2n−

1 ;lin

earin

equality

constrain

tsin

volveon

lyh

1 ,...,hn

ACC’0

16

Euclid

ean

dista

nce

matrix

(ED

M)

D∈

Rn×

nis

ED

Mif

there

arex

1 ,...,xn∈

Rr

s.t.D

ij=‖x

i−

xj ‖

2

ris

calledem

beddin

gdim

ensio

n

[Sch

oen

berg

’35]D

=D

T∈

Rn×

nis

ED

Mw

ithem

bed

din

gdim

ension

riff

•D

ii=

0,

•VD

V�

0,w

here

V=

I−

1n11

T,

•R

ank

VD

V≤

r

ACC’0

17

Min

imum

em

beddin

gdim

ensio

n

find

ED

MD

with

min

imum

embed

din

gdim

ension

satisfying

distan

cebou

nds

min

imize

Rank

VD

V

subject

toD

ii=

0i=

1,...,n

VD

V�

0L

ij≤

Dij≤

Uij

i,j=

1,...,n

.

application

s:

•statistics/p

sychom

etrics(called

multid

imen

sional

scaling)

•ch

emistry

(molecu

larcon

formation

)

ACC’0

18

Exa

ctso

lutio

nm

eth

ods

•sp

ecialcases

with

analytical

solution

svia

SV

D,EV

,e.g

.,

min

imize

Rank

X

subject

to‖X−

A‖≤

b

solution

:su

mof

rlead

ing

dyad

sin

SV

Dof

A,w

here

σr

>b,

σr+

1≤

b

•sp

ecialcases

that

reduce

tocon

vexprob

lems

(e.g

.,[M

esbah

i’97])

•glob

alop

timization

(e.g

.,bran

chan

dbou

nd)

ACC’0

19

Heuristic

solu

tion

meth

ods

•Factorization

meth

ods

(e.g

.,[Iw

asaki’99])

Rank

X≤

riff

there

areF∈

Rm×

r,G∈

Rr×

ns.t.

X=

FG

solvefeasib

ilityprob

lemw

ithvariab

lesF

,G

;use

bisection

onr

•A

nalytic

anti-cen

tering

[David

’94]

use

New

tonm

ethod

inreverse

tom

axim

ize

logbarrier

orpoten

tialfu

nction

ACC’0

110

•A

lternatin

gprojection

s[G

rigoriadis

&B

eran’00]

alternate

betw

een

–projectin

gon

tocon

straint

setC

viacon

vexop

timization

–projectin

gon

toset

ofm

atricesof

rank

rvia

SV

D

PSfrag

replacem

ents

Rank

X≤

r

C

x0

ACC’0

111

Tra

ceheuristic

for

PSD

matrice

s

forX

=X

T�

0,m

inim

izing

traceten

ds

togive

lowran

ksolu

tion[M

esbah

i’97,

Pare

’00]

RM

PTrace

heu

ristic

min

imize

Rank

X

subject

toX∈C

min

imize

TrX

subject

toX∈C

•co

nve

xprob

lem,hen

ceeffi

ciently

solved

•provid

eslo

wer

bound

onm

inran

kob

jective

ACC’0

112

variation:

weigh

tedtrace

min

imization

min

imize

TrW

X

subject

toX∈C

where

W=

WT�

0

ACC’0

113

Log-d

et

heuristic

for

PSD

matrice

s

forX

=X

T�

0,(lo

cally)m

inim

izelog

det(X

+δI)

(δ>

0is

small

constan

tfor

regularizatio

n)

RM

PLog-d

etheu

ristic

min

imize

Rank

X

subject

toX∈C

min

imize

logdet(X

+δI)

subject

toX∈C

•ob

jectiveis

nonco

nve

x(in

fact,con

cave)

•can

use

any

meth

od

tofind

alo

calm

inim

um

ACC’0

114

Idea

behin

dlo

g-d

et

heuristic

Rank

X=

n∑i=

1

1(λi>

0)log

det(X

+δI)=

n∑i=

1

log(λ

i+

δ)

PSfrag

replacem

ents

λi

log(λ

i+

δ)

log

δ 1

ACC’0

115

Itera

tivelin

eariza

tion

meth

od

linearize

(concave)

objective

atX

k�

0:

logdet(X

+δI)≈

logdet(X

k+

δI)

+Tr(X

k+

δI)−

1(X−

Xk )

min

imize

linearized

objective

(acon

vexprob

lem):

Xk+

1=

argmin

X∈C

Tr(X

k+

δI)−

1X

i.e.,

iterativeweigh

tedtrace

min

imization

•w

ithX

0=

I,first

iterationsam

eas

traceheu

ristic

•on

lya

fewiteration

sneed

ed(ab

out

5or

6)

ACC’0

116

Sem

idefinite

em

beddin

g

questio

n:

canwe

extend

trace&

log-det

heu

risticsto

general

(non

square,

non

PSD

)m

atrices?

letX∈

Rm×

n

then

Rank

X≤

riff

there

areY

=Y

T∈

Rm×

m,Z

=Z

T∈

Rn×

n,s.t.

Rank

[

Y0

0Z

]

≤2r,

[

YX

XT

Z

]

�0

thus,

canem

bed

general

(non

PSD

)RM

Pin

a(larger)

PSD

RM

P

ACC’0

117

RM

Pin

em

bedded

PSD

form

min

imize

Rank

X

subject

toX∈C

equivalen

tto

PSD

RM

Pmin

imize

Rank

[

Y0

0Z

]

subject

to

[

YX

XT

Z

]

�0

X∈C,

with

variables

X∈

Rm×

n,Y

=Y

T∈

Rm×

m,Z

=Z

T∈

Rn×

n

cannow

apply

any

meth

od

forsym

metric

PSD

RM

P

ACC’0

118

Tra

ceheuristic

for

genera

lm

atrice

s

min

imize

Tr

[

Y0

0Z

]

subject

to

[

YX

XT

Z

]

�0

X∈C

cansh

owth

isis

equivalen

ttom

inim

ize‖X‖∗

subject

toX∈C,

where

‖X‖∗

=∑

ni=1σ

i (X),

callednucle

arnorm

ofX

,is

dual

ofsp

ectral(m

aximum

singu

larvalu

e)norm

ACC’0

119

Conve

xenve

lope

conve

xenve

lope

off

:C→

Ris

largestcon

vexfu

nction

gs.t.

g(x

)≤

f(x

)for

allx∈

C

PSfrag

replacem

ents

f(x

)

g(x

)

•‘b

est’con

vexlow

erap

proximation

•ep

igraph

ofg

iscon

vexhull

ofep

igraph

off

ACC’0

120

Conve

xenve

lope

ofra

nk

fact:

‖X‖∗

iscvx

envelop

eof

Rank

Xon

{X∈

Rm×

n|‖X‖≤

1}

conclu

sions:

•trace

heu

risticm

inim

izesco

nve

xenve

lope

ofran

k(i.e

.,th

ebest

convex

approxim

ationto

rank)

overball

•hen

ce,heu

risticprovid

eslow

erbou

nd

onob

jective

•provid

esth

eoreticalsu

pport

foruse

oftrace/n

uclear

norm

heu

ristic

ACC’0

121

Maxim

um

sparsity

pro

ble

m

nuclear

norm

heu

risticfor

diagon

alX

becom

es

min

imize

‖x‖1

subject

tox∈C

•well-kn

own

`1

heu

risticfor

findin

gsp

arsesolu

tions

•used

inLA

SSO

meth

ods

instatistics

[Tib

shiran

i’94],

signal

decom

position

bybasis

pursu

it[D

onoh

o’96],

...

•‖x‖1

iscon

vexen

velope

ofC

ard(x

)on

{x|‖x‖∞≤

1}

•trace/n

uclear

norm

heu

risticis

extension

of`1

heu

risticto

matrix

case

ACC’0

122

Log-d

et

heuristic

for

genera

lm

atrice

s

min

imize

logdet

([

Y0

0Z

]

+δI

)

subject

to

[

YX

XT

Z

]

�0

X∈C

cansh

owth

isis

equivalen

tto

min

imize

∑

ni=1log

(σi (X

)+

δ)su

bject

toX∈C

canlin

earizeas

before

toob

tainiteration

sin

X,Y

,Z

ACC’0

123

Itera

tive`1

heuristic

for

maxim

um

sparsity

pro

ble

m

log-det

heu

risticfor

maxim

um

sparsity

problem

yields

min

imize

∑

i log(|x

i |+δ)

subject

tox∈C.

iterativelin

earization/m

inim

izationyield

s

x(k

+1)=

argmin

x∈C

n∑i=

1

w(k

)i|x

i |,w

(k)

i=

1

|x(k

)i|+

δ

•each

stepis

weig

hte

d`1

norm

min

imization

•w

hen

x(k

)i

small,

weigh

tin

next

stepis

large;hen

ce,sm

allen

triesin

x

arepush

edtow

ards

zero(su

bject

tox∈C)

ACC’0

124

Exa

mple

:m

inim

um

ord

ersyste

mre

aliza

tion

with

step

resp

onse

constra

ints

find

min

imum

order

systemth

atsatisfi

esli≤

si≤

ui ,

i=

1,...,16

•n

=16

•trace/n

uclear

norm

heu

risticyield

sran

k5

•log-d

etheu

risticcon

vergesin

5step

s,yield

sran

k4

05

10

15

20

25

30

35

−0

.2 0

0.2

0.4

0.6

0.8 1

1.2

PSfrag

replacem

ents

stepresp

onse

(solid

)an

dsp

ecs(d

ashed

)

t

ACC’0

125

11.5

22.5

33.5

44.5

5−

10

−9

−8

−7

−6

−5

−4

−3

−2

−1 0

PSfrag

replacem

ents log of non-zero σis

iterations

ACC’0

126

Conclu

sions

•RM

Pis

diffi

cult

non

convex

problem

,w

ithm

any

application

s

•m

aximum

sparsity

problem

issp

ecialcase

•trace

and

log-det

heu

risticsfor

PSD

RM

Pcan

be

extended

togen

eralcase,

viasem

idefi

nite

embed

din

g

•gen

eralizationof

trace

heu

risticto

general

matrices

isnucle

arnorm

,w

hich

iscon

vexen

velope

ofran

k

ACC’0

127