Extremum Properties of

Orthogonal Quotients Matrices

By

Achiya Dax

Hydrological Service, Jerusalem , Israel

e-mail: dax20@water.gov.il

The Eckart – Young Theorem ( 1936 )

says that the “truncated SVD” matrix

Ak = Uk Dk VkT

solves the least norm problem

minimize F ( B ) = || A - B || F

2

subject to rank ( B ) k

( Also called the Schmidt-Mirsky Theorem. )

Ky Fan’s Maximum Principle ( 1949 )

S a symmetric positive semi-definite n x n matrix

Yk the set of orthogonal n x k matrices

1 + … + k = max { trace ( YkT S Yk ) | YkYk }

Solution is obtained for the Spectral matrix

Vk = [v1 , v2 , … , vk] .

giving the sum of the largest k eigenvalues.

Outline

* The Eckart – Young Theorem

and Orthogonal Quotients matrices.

* The Orthogonal Quotients Equality.

* The Symmetric Quotients Equality

and Ky Fan’s extremum principles.

* An Extended Extremum Principle.

The Singular Value Decomposition

A = U V T

= diag { 1 , 2 , … , p } , p = min { m , n }

U = [u1 , u2 , … , up] , U T

U = I

V = [v1 , v2 , … , vp] , VT

V = I

A V = U AT U = V

A vj = j uj , AT uj = j vj j = 1, … , p .

Low - Rank Approximations

Ak = Uk Dk VkT

Dk = diag { 1 , 2 , … , k } ,

Uk = [u1 , u2 , … , uk] , UkT Uk = I

Vk = [v1 , v2 , … , vk] , VkT Vk = I

The Eckart – Young Theorem ( 1936 )

says that the “truncated SVD” matrix

Ak = Uk Dk VkT

solves the least norm problem

minimize F ( B ) = || A - B || F

2

subject to rank ( B ) k

( Also called the Schmidt-Mirsky Theorem. )

Rank - k Matrices

B = Xk Rk YkT

where

Rk is a k x k matrix

Xk = [x1 , x2 , … , xk] , XkT Xk = I

Yk = [y1 , y2 , … , yk] , YkT Yk = I

The Eckart – Young Problem

can be rewritten as

minimize F ( B ) = || A - Xk Rk YkT || F

2

subject to XkT Xk = I and Yk

T Yk = I .

Theorem 1 : Given a pair of orthogonal

matrices, Xk and Yk , the related

“Orthogonal Quotients Matrix”

XkT

A Yk = ( xiTAyj )

solves the problem

minimize F ( Rk ) = || A - Xk Rk YkT || F

2

Notation:

Xk - denotes the set of all real m x k

orthogonal matrices Xk ,

Xk = [x1 , x2 , … , xk] , XkT Xk = I

Yk - denotes the set of all real n x k

orthogonal matrices Yk ,

Yk = [y1 , y2 , … , yk] , YkT Yk = I

Corollary 1 : The Eckart – Young Problem can be rewritten as

minimize F ( Xk, Yk ) = || A - Xk Rk YkT

|| F

2

subject to XkXk and YkYk ,

where Rk is the “Orthogonal Quotients Matrix”

Rk = XkT

A Yk .

The Orthogonal Quotients Equality

For any pair of orthogonal matrices,

XkXk and YkYk ,

|| A - Xk Rk YkT

|| F

2 = || A || F 2 - || Rk || F

2

where Rk is the orthogonal quotients matrix

Rk = XkT

A Yk .

Corollary : The Eckart – Young Problem

minimize F ( Xk, Yk ) = || A - Xk Rk YkT

|| F

2

subject to XkXk and YkYk .

is equivalent to

maximize || XkT

A Yk || F

2

subject to XkXk and YkYk ,

and the SVD matrices Uk and Vk solves both problems,

giving the optimal value of 12

+ 22 + … + k

2 .

Question : Is the related minimum problem

minimize || (Xk)T A Yk || F

2

subject to XkXk and YkYk ,

solvable ?

Using the Orthogonal Quotients Equality

the last problem takes the form

maximize F ( Xk, Yk ) = || A - Xk Rk YkT

|| F

2

subject to XkXk and YkYk .

Note that the more general problem

maximize F ( B ) = || A - B || F

2

subject to rank ( B ) k

is not solvable .

Returning to symmetric matrices

How we extend the

Orthogonal Quotients Equality

to symmetric matrices ?

The Spectral Decomposition

S = ( Sij ) a symmetric positive semi-definite n x n matrix

With eigenvalues n

and eigenvectors v1 , v2 , … , vn

S vj = j vj , j = 1, … , n . S V = V D

V = [v1 , v2 , … , vn] , VT V = V VT = I

D = diag { 1 , 2 , … , n }

S = V D VT = j vj vjT

Recall that

Rayleigh Quotient Matrices

Sk = YkT S Yk

play important role in

Ky Fan’s Extremum Principles .

Ky Fan’s Maximum Principle

S a symmetric positive semi-definite n x n matrix

Yk the set of orthogonal n x k matrices

1 + … + k = max { trace ( YkT S Yk ) | YkYk }

Solution is obtained for the Spectral matrix

Vk = [v1 , v2 , … , vk] .

which is related to the largest k eigenvalues.

Ky Fan’s Minimum Principle

S a symmetric n x n matrix .

Yk the set of orthogonal n x k matrices .

n - k+1 + … + n = min { trace ( YkT S Yk ) | YkYk }

Solution is obtained for the Spectral matrix

Vk = [vn-k+1 , … , vn] ,

which is related to the smallest k eigenvalues.

Question : Can we formulate the

Symmetric Quotients Equality

in terms of

trace ( Sk ) = trace ( YkT S Yk ) = yj

T S yj

The Symmetric Quotients Equality

S a symmetric n x n matrix

YkYk an orthogonal n x k matrix

Sk = YkT

S Yk the related “Rayleigh quotient matrix”

trace ( S - Yk Sk YkT

) = trace ( S ) - trace( Sk )

Corollary 1 : Ky Fan’s maximum problem

maximize trace (Yk S YkT )

subject to YkYk ,

is equivalent to

minimize trace ( S - Yk Sk YkT

)

subject to YkYk .

The Spectral matrix Vk = [v1 , v2 , … , vk]

solves both problems,

giving the optimal value of 1 + … + k .

Recall that : The Eckart – Young Problem

minimize F ( Xk, Yk ) = || A - Xk Rk YkT

|| F

2

subject to XkXk and YkYk .

is equivalent to

maximize || XkT

A Yk || F

2

subject to XkXk and YkYk ,

and the SVD matrices Uk and Vk solves both problems,

giving the optimal value of 12

+ 22 + … + k

2 .

Corollary 2 : Ky Fan’s minimum problem

minimize trace (Yk S YkT )

subject to YkYk ,

is equivalent to

maximize trace ( S - YkT

Sk Yk )

subject to YkYk .

The matrix Vk = [vn-k+1 , … , vn]

solves both problems,

giving the optimal value of n-k+1 + … + n .

Extended Exremum Principles

Can we extend these extremums

from eigenvalues of symmetric matrices

to singular values of rectangular matrices ?

Notation: 1 m* m , 1 n* n ,

Xm* - denotes the set of all real m x m*

orthogonal matrices Xm* ,

Xm* = [x1 , x2 , … , xm*] , Xm*T Xm* = I

Yn* - denotes the set of all real n x n*

orthogonal matrices Yn* ,

Yn* = [y1 , y2 , … , yn*] , Yn*T Y* = I

Notations :

Given Xm* Xm* and Yn* Ym* ,

the m* x n* matrix

( Xm*)TA Yn* = ( xiTAyj )

is called “Orthogonal Quotients Matrix” .

Notations :

The singular values of the

Orthogonal Quotients Matrix

( Xm*)TA Yn* = ( xiTAyj )

are denoted as

1 2 … k 0 , where

k = min { m*, n* } .

Questions :

Which choice of orthogonal matrices

Xm* Xm* and Yn* Ym* ,

maximizes (or minimizes ) the sum

(1)p + (2)p + … + (k)p

where p > 0 is a given positive constant .

An Extended Maximum Principle : The SVD matrices

Um* = [u1 , u2 , … , um*] and Vn* = [v1 , v2 , … , vn*]

solve the problem

maximize F ( Xm* , Yn* ) = (1)p + (2)p + … + (k)p

subject to Xm* Xm* and Yn* Yn* ,

for any positive power p > 0 ,

giving the optimal value of (1)p + (2)p + … + (k)p .

An Extended Maximum Principle

That is, for any positive power p > 0 ,

(1)p + (2)p + … + (k)p =

max{ (1)p + (2)p +…+ (k)p | Xm* Xm* and Yn* Yn*},

and the maximal value is attained for the matrices

Um* = [u1 , u2 , … , um*] and Vn* = [v1 , v2 , … , vn*] .

The proof

is based on “rectangular” versions of

Cauchy Interlace Theorem

and

Poincare Separation Theorem .

Corollary 1 : When p = 1 the SVD matrices

Um* = [u1 , u2 , … , um*] and Vn* = [v1 , v2 , … , vn*]

solve the “Rectangular Ky Fan problem”

maximize F ( Xm* , Yn* ) = 1 + 2 + … + k

subject to Xm* Xm* and Yn* Yn* ,

giving the optimal value of

1 + 2 + … + k .

Corollary 2 : When p = 1 and m* = n* = k

the SVD matrices

Uk = [u1 , u2 , … , uk] and Vk = [v1 , v2 , … , vk]

solve the maximum trace problem

maximize F ( Xk , Yk ) = trace ( (Xk)T

A Yk )

subject to Xk Xk and Yk Yk ,

giving the optimal value of

1 + 2 + … + k .

* See also Horn & Johnson, “Topics in Matrix Analysis”, p. 195.

Corollary 3 : When p = 2 the SVD matrices

Um* = [u1 , u2 , … , um*] and Vn* = [v1 , v2 , … , vn*]

solve the “rectangular Eckart – Young problem”

maximize F ( Xm* , Yn* ) = || (Xm*)T

A Yn* || F

2

subject to Xm* Xm* and Yn* Yn* ,

giving the optimal value of

(1)2 + (2)2 + … + (k)2 .

An Extended Minimum Principle

Question : Can we prove a similar

minimum principle ?

Answer : Yes, but the solution matrices

are more complicated .

An Extended Minimum Principle : Here we consider the problem

minimize F( Xm* , Yn* ) = (1)p + (2)p + … + (k)p

subject to Xm* Xm* and Yn* Yn* ,

for any positive power p > 0 .

The solution matrices are obtained by deleting some

columns from the SVD matrices

U = [u1 , u2 , … , um] and V = [v1 , v2 , … , vn] .

Summary

* The Eckart – Young Theorem

and Orthogonal Quotients matrices.

* The Orthogonal Quotients Equality.

* The Symmetric Quotients Equality

and Ky Fan’s extremum principles.

* An Extended Extremum Principle.

The END

Thank You