Tensor Decompositions, Matrix Completion and Singular Valuessites.lsa.umich.edu/hderksen/wp-content/uploads/sites/614/2018/09/… · Tensor Decompositions, Matrix Completion and Singular

Tensor Decompositions, Matrix Completion andSingular Values

Harm Derksen

University of Michigan

ICERM, Computational Nonlinear AlgebraJune 3, 2014

Harm Derksen Tensors, LRMC and Singular Values

Tensor Product Spaces

F a fieldV (i) ∼= Fni F-vector space for i = 1, 2, . . . , dV = V (1) ⊗ V (2) ⊗ · · · ⊗ V (d) ∼= Fn1×···×nd tensor product space

Definition

A pure tensor is a tensor of the form v (1) ⊗ v (2) ⊗ · · · ⊗ v (d)

(v (i) ∈ V (i)).

Problem

Write a given tensor as a sum of the smallest number of puretensors.

(for F = R,C: Canonical Polyadic (CP) decompositions,PARAFAC, CANDECOMP)


Tensor Product Spaces

F a fieldV (i) ∼= Fni F-vector space for i = 1, 2, . . . , dV = V (1) ⊗ V (2) ⊗ · · · ⊗ V (d) ∼= Fn1×···×nd tensor product space

Definition

A pure tensor is a tensor of the form v (1) ⊗ v (2) ⊗ · · · ⊗ v (d)

(v (i) ∈ V (i)).

Problem

Write a given tensor as a sum of the smallest number of puretensors.

(for F = R,C: Canonical Polyadic (CP) decompositions,PARAFAC, CANDECOMP)


Applications

I psychometrics

I chemometrics

I algebraic complexity theory

I signal processing

I numerical linear algebra

I computer vision

I numerical analysis

I data mining

I graph analysis

I neuroscience

I economics/finance


Application: Fluorescence Spectroscopy

We have p samples of mixtures of unknown chemical compounds.Every mixture is excited with light of m different wavelengths.Light of n wavelengths is emitted from the mixture . Measuringthe intensities, one obtains an m × n excitation-emisson matrix forevery sample. This yields a p ×m × n matrix, which is a 3-waytensor T .Every chemical compound corresponds to a rank 1 tensor. Bywriting T as the sum of r = rank(T ) pure tensors, we candistinguish r chemical compounds and find the excitation-emissionmatrix for each of them.


Application:Fluorescence Spectroscopy(Image: Lei Li, Andrew Barron)


Tensor rank

Definition

The rank of a tensor T ∈ V is the smallest nonnegative integer rsuch that we can write T as a sum of r pure tensors.

For example,

T = e1 ⊗ e1 ⊗ e1 + e1 ⊗ e2 ⊗ e2 + e2 ⊗ e1 ⊗ e2 + e2 ⊗ e2 ⊗ e1

has rank 2 because

T = 12(e1+e2)⊗(e1+e2)⊗(e1+e2)+ 1

2(e1−e2)⊗(e1−e2)⊗(e1−e2)

If d = 2 and V = V (1) ⊗ V (2) ∼= Rd1×d2 then the tensor rank isjust the rank of a matrix.


Tensor rank

Definition


For example,

T = e1 ⊗ e1 ⊗ e1 + e1 ⊗ e2 ⊗ e2 + e2 ⊗ e1 ⊗ e2 + e2 ⊗ e2 ⊗ e1

has rank 2 because

T = 12(e1+e2)⊗(e1+e2)⊗(e1+e2)+ 1

2(e1−e2)⊗(e1−e2)⊗(e1−e2)



Tensor rank

Definition


For example,

T = e1 ⊗ e1 ⊗ e1 + e1 ⊗ e2 ⊗ e2 + e2 ⊗ e1 ⊗ e2 + e2 ⊗ e2 ⊗ e1

has rank 2 because

T = 12(e1+e2)⊗(e1+e2)⊗(e1+e2)+ 1

2(e1−e2)⊗(e1−e2)⊗(e1−e2)



Tensor rank

Definition


For example,

T = e1 ⊗ e1 ⊗ e1 + e1 ⊗ e2 ⊗ e2 + e2 ⊗ e1 ⊗ e2 + e2 ⊗ e2 ⊗ e1

has rank 2 because

T = 12(e1+e2)⊗(e1+e2)⊗(e1+e2)+ 1

2(e1−e2)⊗(e1−e2)⊗(e1−e2)



Application: Algebraic Complexity Theory

V = Matn,n(F)⊗Matn,n(F)⊗Matn,n(F)

Tn =∑n

i ,j ,k=1 ei ,j ⊗ ej ,k ⊗ ek,i

rank(Tn) is the number of multiplications needed to multiply twon × n matrices. Clearly rank(Tn) ≤ n3.

Theorem

If rank(Tm) ≤ k , then rank(Tn) = O(nlogm(k)) and two n × nmatrices can be multiplied using O(nlogm(k)) arithmetic operations.



V = Matn,n(F)⊗Matn,n(F)⊗Matn,n(F)

Tn =∑n

i ,j ,k=1 ei ,j ⊗ ej ,k ⊗ ek,i

rank(Tn) is the number of multiplications needed to multiply twon × n matrices. Clearly rank(Tn) ≤ n3.

Theorem

If rank(Tm) ≤ k , then rank(Tn) = O(nlogm(k)) and two n × nmatrices can be multiplied using O(nlogm(k)) arithmetic operations.



Theorem (Strassen 1969)

rank(T2) ≤ 7, so rank(Tn) = O(nlog2(7)) = O(n2.8073...).

Theorem (Williams 2012)

rank(Tn) = O(n2.3727)

Theorem (Masseranti, Raviolo 2013)

rank(Tn) ≥ 3n2 − 2√

2n3/2 − 3n.


Low Rank Matrix Completion

Problem

Given a partially filled matrix, complete the matrix such that theresulting matrix has the smallest possible rank.

For example −2 · ·2 3 ·· 6 2

can be completed to a rank 1 matrix−2 −3 −1

2 3 14 6 2



Problem


For example −2 · ·2 3 ·· 6 2

can be completed to a rank 1 matrix−2 −3 −12 3 14 6 2



Problem


For example −2 · ·2 3 ·· 6 2

can be completed to a rank 1 matrix−2 −3 −1

2 3 14 6 2


Application: The Netflix Problem

The DVD rental company Netflix has 480,189 users, and 17,770movies. The user ratings for every user can be put in a480, 189× 17, 770 matrix A = (ai ,j) for which only few entries areknown (since most users have only seen a fraction of the 17,770movies).

Presumably, the rank of the matrix A is low. Using Low rankmatrix completion, Netflix can predict whether users like a moviethey have seen, and will recommend movies to the users.


Reduction LRMC to Tensor Rank

A is n ×m matrixare allowed to change entries in positions (i1, j1), . . . , (is , js).mrank(A) minimal possible rank

Define

T =s∑

k=1

eik ⊗ ejk ⊗ ek + A⊗ es+1 ∈ Cn ⊗ Cm ⊗ Cs+1

Theorem (D.)

rank(T ) = mrank(A) + s


Reduction LRMC to Tensor Rank

A is n ×m matrixare allowed to change entries in positions (i1, j1), . . . , (is , js).mrank(A) minimal possible rank

Define

T =s∑

k=1

eik ⊗ ejk ⊗ ek + A⊗ es+1 ∈ Cn ⊗ Cm ⊗ Cs+1

Theorem (D.)

rank(T ) = mrank(A) + s


Example

A =

(1 t· 1

)For T = e2 ⊗ e1 ⊗ e1 + e1 ⊗ (e1 + te2)⊗ e2 + e2 ⊗ e2 ⊗ e2 we haverank(T ) = 1 + mrank(A).

mrank(A) = 1 if t 6= 0 and mrank(A) = 2 if t = 0.

If t = 0, then T = e2 ⊗ e1 ⊗ e1 + e1 ⊗ e1 ⊗ e2 + e2 ⊗ e2 ⊗ e2 hasrank 3.

If t 6= 0 then rank(T ) = 2 and

T = e2 ⊗ e1 ⊗ (e1 − t−1e2) + (e1 + t−1e2)⊗ (e1 + te2)⊗ e2


Example

A =

(1 t· 1





T = e2 ⊗ e1 ⊗ (e1 − t−1e2) + (e1 + t−1e2)⊗ (e1 + te2)⊗ e2


Example

A =

(1 t· 1





T = e2 ⊗ e1 ⊗ (e1 − t−1e2) + (e1 + t−1e2)⊗ (e1 + te2)⊗ e2


Example

A =

(1 t· 1





T = e2 ⊗ e1 ⊗ (e1 − t−1e2) + (e1 + t−1e2)⊗ (e1 + te2)⊗ e2


Rank Minimization Problem (RM)

Problem (Rank Minimization)

Given A,B1, . . . ,Bs ∈ Matp,q(F), minimizerank(A + x1B1 + · · ·+ xsBs).

LRMC is a special case of RM where Bk = eik ,jk .

Theorem (D.)

The Tensor Rank problem can be reduced to RM and RM can bereduced to LRMC.

For example, suppose that T = (ti ,j ,k)2i ,j ,k=1 is a 2× 2× 2 tensor.


Rank Minimization Problem (RM)

Problem (Rank Minimization)

Given A,B1, . . . ,Bs ∈ Matp,q(F), minimizerank(A + x1B1 + · · ·+ xsBs).

LRMC is a special case of RM where Bk = eik ,jk .

Theorem (D.)

The Tensor Rank problem can be reduced to RM and RM can bereduced to LRMC.

For example, suppose that T = (ti ,j ,k)2i ,j ,k=1 is a 2× 2× 2 tensor.


t1,1,1 t1,2,1 a1,1 a1,2 a1,3 0 0 0 0 0 0 0 0 0 0 0 0 0 0t2,1,1 t2,2,1 a2,1 a2,2 a2,3 0 0 0 0 0 0 0 0 0 0 0 0 0 0

0 0 1 0 0 λ1,1 0 0 0 0 0 0 0 0 0 0 0 0 00 0 0 1 0 0 λ1,2 0 0 0 0 0 0 0 0 0 0 0 00 0 0 0 1 0 0 λ1,3 0 0 0 0 0 0 0 0 0 0 0

b1,1 b2,1 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0b1,2 b2,2 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0b1,3 b2,3 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 00 0 0 0 0 0 0 0 t1,1,2 t1,2,2 a1,1 a1,2 a1,3 0 0 0 0 0 00 0 0 0 0 0 0 0 t2,1,2 t2,2,2 a2,1 a2,2 a2,3 0 0 0 0 0 00 0 0 0 0 0 0 0 0 0 1 0 0 λ2,1 0 0 0 0 00 0 0 0 0 0 0 0 0 0 0 1 0 0 λ2,2 0 0 0 00 0 0 0 0 0 0 0 0 0 0 0 1 0 0 λ2,3 0 0 00 0 0 0 0 0 0 0 b1,1 b2,1 0 0 0 1 0 0 0 0 00 0 0 0 0 0 0 0 b1,2 b2,2 0 0 0 0 1 0 0 0 00 0 0 0 0 0 0 0 b1,3 b2,3 0 0 0 0 0 1 0 0 00 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 a1,1 0 00 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 a2,1 0 00 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 b1,1 0 00 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 b2,1 0 00 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 a1,2 00 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 a2,2 00 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 b1,2 00 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 b2,2 00 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 a1,30 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 a2,30 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 b1,30 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 b2,3

.

The smallest rank over all ai ,j , bi ,j , λi ,j (1 ≤ i ≤ 2, 1 ≤ j ≤ 3) is12 + rank(T ).


Motivation: compressed sensing and convex relaxation

For x ∈ Rn, its sparsity is measured by ‖x‖0 = #{i | xi 6= 0}.

Problem

Given A ∈ Matm,n(R), b ∈ Rm, find a solution x ∈ Rn for Ax = bwith ‖x‖0 minimal (a sparsest solution).

But, ‖ · ‖0 is not convex and this optimization problem is difficult,

so instead we consider:

Problem (Basis Pursuit)

Given A ∈ Matm,n(R), b ∈ Rm, find a solution x ∈ Rn for Ax = bwith ‖x‖1 minimal.

Basis Pursuit can be solved by linear programming and is generallyfast. Under reasonable assumptions, Basis Pursuit also gives thesparsest solutions (Candes-Tao, Donoho).


Motivation: compressed sensing and convex relaxation

For x ∈ Rn, its sparsity is measured by ‖x‖0 = #{i | xi 6= 0}.

Problem

Given A ∈ Matm,n(R), b ∈ Rm, find a solution x ∈ Rn for Ax = bwith ‖x‖0 minimal (a sparsest solution).

But, ‖ · ‖0 is not convex and this optimization problem is difficult,so instead we consider:

Problem (Basis Pursuit)

Given A ∈ Matm,n(R), b ∈ Rm, find a solution x ∈ Rn for Ax = bwith ‖x‖1 minimal.

Basis Pursuit can be solved by linear programming and is generallyfast. Under reasonable assumptions, Basis Pursuit also gives thesparsest solutions (Candes-Tao, Donoho).


Nuclear Norm

V (i) Hilbert space. Instead of the CP model, we consider:

Problem (Convex Decomposition)

Given a tensor T , write T =∑r

i=1 vi for some r and some puretensors v1, . . . , vr such that

∑ri=1 ‖vi‖2 is minimal.

Finding a convex decomposition seems to be easier than finding aCP decomposition. Heuristically, we expect convex decompositionsto give a CP decomposition or at least a low rank decomposition.

Definition (Lim-Comon)

The nuclear norm ‖T‖? is the smallest value of∑r

i=1 ‖vi‖2 whereT =

∑ri=1 vi and v1, . . . , vr are pure tensors.

For some tensors we know the nuclear norm but not the rank.


Nuclear Norm

V (i) Hilbert space. Instead of the CP model, we consider:

Problem (Convex Decomposition)

Given a tensor T , write T =∑r

i=1 vi for some r and some puretensors v1, . . . , vr such that

∑ri=1 ‖vi‖2 is minimal.

Finding a convex decomposition seems to be easier than finding aCP decomposition. Heuristically, we expect convex decompositionsto give a CP decomposition or at least a low rank decomposition.

Definition (Lim-Comon)

The nuclear norm ‖T‖? is the smallest value of∑r

i=1 ‖vi‖2 whereT =

∑ri=1 vi and v1, . . . , vr are pure tensors.

For some tensors we know the nuclear norm but not the rank.


Spectral Norm

Definition

The spectral norm is defined by

‖T‖σ = max{|〈T , v〉| | v pure tensor with ‖v‖2 = 1}.

The spectral norm is dual to the nuclear norm, in particular

|〈T ,S〉| ≤ ‖T‖?‖S‖σ

for all tensors S ,T .


Example: Determinant Tensor

Consider the tensor

Dn =∑σ∈Sn

sgn(σ)eσ(1) ⊗ eσ(2) ⊗ · · · ⊗ eσ(n) ∈ Cn ⊗ · · · ⊗ Cn.

Clearly rank(Dn) ≤ n!, actually( nbn/2c

)≤ rank(Dn) ≤ (56)bn/3cn!.

‖Dn‖σ = max{| det(v1v2 · · · vn)| | ‖v1‖2 = · · · = ‖vn‖2 = 1} = 1

by Hadamard’s inequality.

‖Dn‖? = ‖Dn‖?‖Dn‖σ ≥ 〈Dn,Dn〉 = n!, so

Theorem (D.)

‖Dn‖? = n!


Example: Determinant Tensor

Consider the tensor

Dn =∑σ∈Sn

sgn(σ)eσ(1) ⊗ eσ(2) ⊗ · · · ⊗ eσ(n) ∈ Cn ⊗ · · · ⊗ Cn.

Clearly rank(Dn) ≤ n!, actually( nbn/2c

)≤ rank(Dn) ≤ (56)bn/3cn!.

‖Dn‖σ = max{| det(v1v2 · · · vn)| | ‖v1‖2 = · · · = ‖vn‖2 = 1} = 1

by Hadamard’s inequality.

‖Dn‖? = ‖Dn‖?‖Dn‖σ ≥ 〈Dn,Dn〉 = n!, so

Theorem (D.)

‖Dn‖? = n!


Example: Permanent Tensor

Pn =∑

σ∈Sn eσ(1) ⊗ eσ(2) ⊗ · · · ⊗ eσ(n) ∈ Cn ⊗ · · · ⊗ Cn.

‖Pn‖σ = max{| perm(v1v2 · · · vn)| | ‖v1‖2 = · · · = ‖vn‖2 = 1}

Theorem (Carlen, Lieb and Moss, 2006)

max{perm(v1v2 · · · vn) | ‖v1‖2 = · · · = ‖vn‖2 = 1} = n!/nn/2

‖Pn‖? =nn/2

n!‖Pn‖?‖Pn‖σ ≥

nn/2

n!〈Pn,Pn〉 = nn/2.



Pn =∑

σ∈Sn eσ(1) ⊗ eσ(2) ⊗ · · · ⊗ eσ(n) ∈ Cn ⊗ · · · ⊗ Cn.

‖Pn‖σ = max{| perm(v1v2 · · · vn)| | ‖v1‖2 = · · · = ‖vn‖2 = 1}

Theorem (Carlen, Lieb and Moss, 2006)

max{perm(v1v2 · · · vn) | ‖v1‖2 = · · · = ‖vn‖2 = 1} = n!/nn/2

‖Pn‖? =nn/2

n!‖Pn‖?‖Pn‖σ ≥

nn/2

n!〈Pn,Pn〉 = nn/2.



Theorem (Glynn 2010)

Pn =1

2n−1

∑δ

(∏ni=1 δi

)(∑n

i=1 δiei )⊗ · · · ⊗ (∑n

i=1 δiei )

where δ runs over {1} × {−1, 1}n−1.

In particular,( nbn/2c

)≤ rank(Pn) ≤ 2n−1 and ‖Pn‖? ≤ nn/2, so

Theorem (D.)

‖Pn‖? = nn/2



Theorem (Glynn 2010)

Pn =1

2n−1

∑δ

(∏ni=1 δi

)(∑n

i=1 δiei )⊗ · · · ⊗ (∑n

i=1 δiei )

where δ runs over {1} × {−1, 1}n−1.

In particular,( nbn/2c

)≤ rank(Pn) ≤ 2n−1 and ‖Pn‖? ≤ nn/2, so

Theorem (D.)

‖Pn‖? = nn/2


t-Orthogonality

Definition

Pure tensors v1, v2, . . . , vr are t-orthogonal if

r∑i=1

|〈vi ,w〉|2/t ≤ 1

for every pure tensor w with ‖w‖2 = 1.

1-orthogonal ⇔ orthogonalIf t > s then t-orthogonal ⇒ s-orthogonal.

Theorem (D.)

If v1, . . . , vr ∈ V are t-orthogonal, then r ≤ dim(V )1/t .


t-Orthogonality

Definition


r∑i=1

|〈vi ,w〉|2/t ≤ 1



Theorem (D.)



t-Orthogonality

Definition


r∑i=1

|〈vi ,w〉|2/t ≤ 1



Theorem (D.)



Horizontal and Vertical Tensor Product

Theorem (“horizontal tensor product”, D.)

If v1, . . . , vr are t-orthogonal, and w1, . . . ,wr are s-orthogonal,then v1 ⊗ w1, . . . , vr ⊗ wr are (s + t)-orthogonal.

If V = V (1) ⊗ · · · ⊗ V (d) and W = W (1) ⊗ · · · ⊗W (d), then

V �W := (V (1) ⊗W (1))⊗ · · · ⊗ (V (d) ⊗W (d)).

Theorem (“vertical tensor product”, D.)

If v1, v2, . . . , vr ∈ V and w1, . . . ,ws ∈W are t-orthogonal, then{vi � wj | 1 ≤ i ≤ r , 1 ≤ j ≤ s} are t-orthogonal.


Horizontal and Vertical Tensor Product

Theorem (“horizontal tensor product”, D.)

If v1, . . . , vr are t-orthogonal, and w1, . . . ,wr are s-orthogonal,then v1 ⊗ w1, . . . , vr ⊗ wr are (s + t)-orthogonal.

If V = V (1) ⊗ · · · ⊗ V (d) and W = W (1) ⊗ · · · ⊗W (d), then

V �W := (V (1) ⊗W (1))⊗ · · · ⊗ (V (d) ⊗W (d)).

Theorem (“vertical tensor product”, D.)

If v1, v2, . . . , vr ∈ V and w1, . . . ,ws ∈W are t-orthogonal, then{vi � wj | 1 ≤ i ≤ r , 1 ≤ j ≤ s} are t-orthogonal.


The Diagonal Singular Value Decomposition

Definition

Suppose that (?) : T =∑r

i=1 λivi such that λ1 ≥ · · · ≥ λr > 0and v1, . . . , vr are 2-orthogonal pure tensors of unit length, then(?) is called a diagonal singular value decomposition of T (DSVD).

If d = 2 (tensor product of 2 spaces) then the DSVD is the usualsingular value decomposition. For d > 2, the DSVD is differentfrom the Higher Order Singular Value Decomposition defined byDe Lathauer, De Moor, and Vandewalle. Not every tensor has aDSVD.



Theorem (D.)

If T has a DSVD then

‖T‖? =∑i

λi , ‖T‖2 =

√∑i

λ2i , ‖T‖σ = λ1

Theorem (D.)

If λ1 > λ2 > · · · > λr then the DSVD is unique.

Theorem (D.)

If v1, . . . , vr are t-orthogonal with t > 2, then the DSVD is unique.



Theorem (D.)

If T has a DSVD then

‖T‖? =∑i

λi , ‖T‖2 =

√∑i

λ2i , ‖T‖σ = λ1

Theorem (D.)

If λ1 > λ2 > · · · > λr then the DSVD is unique.

Theorem (D.)

If v1, . . . , vr are t-orthogonal with t > 2, then the DSVD is unique.


Example: Matrix Multiplication Tensor

e1, . . . , en ∈ Cn are orthogonale1 ⊗ e1, . . . , en ⊗ en ∈ Cn ⊗ Cn are 2-orthogonal

e1 ⊗ e1 ⊗ 1, . . . , en ⊗ en ⊗ e1 ∈ Cn ⊗ Cn ⊗ C are 2-orthogonale1 ⊗ 1⊗ e1, . . . , en ⊗ 1⊗ en ∈ Cn ⊗ C⊗ Cn are 2-orthogonal1⊗ e1 ⊗ e1, . . . , 1⊗ en ⊗ en ∈ C⊗ Cn ⊗ Cn are 2-orthogonal

Using vertical tensor product, we get

{(ei ⊗ ej)⊗ (ej ⊗ ek)⊗ (ek ⊗ ei ) | 1 ≤ i , j , k ≤ n}

are 2-orthogonal.







are 2-orthogonal.







are 2-orthogonal.



Theorem (D.)

The matrix multiplication tensor

Tn =n∑

i ,j ,k=1

ei ,j ⊗ ej ,k ⊗ ek,i

is a DSVD.

The singular values of Tn are

1, 1, . . . , 1︸︷︷︸n3

In particular,

‖Tn‖? =n3∑i=1

1 = n3.


Example: Discrete Fourier Transform

DefineFn =

∑1≤i,j,k≤n

i+j+k≡0 mod n

ei ⊗ ej ⊗ ek

This tensor is related to the multiplication of univariatepolynomials. Clearly rank(Fn) ≤ n2 and ‖Fn‖? ≤ n2.

Discrete Fourier Transform (DFT):

Fn =n∑

j=1

√n( 1√

n

∑ni=1 ζ

ijei )⊗ ( 1√n

∑ni=1 ζ

ijei )⊗ ( 1√n

∑ni=1 ζ

ijei ).

where ζ = eπi/n. This is the unique DSVD of Fn. So the singularvalues are

√n, . . . ,

√n (n times), rank(Fn) = n and ‖Fn‖? = n

√n.


Example: Discrete Fourier Transform

DefineFn =

∑1≤i,j,k≤n

i+j+k≡0 mod n

ei ⊗ ej ⊗ ek

This tensor is related to the multiplication of univariatepolynomials. Clearly rank(Fn) ≤ n2 and ‖Fn‖? ≤ n2.

Discrete Fourier Transform (DFT):

Fn =n∑

j=1

√n( 1√

n

∑ni=1 ζ

ijei )⊗ ( 1√n

∑ni=1 ζ

ijei )⊗ ( 1√n

∑ni=1 ζ

ijei ).

where ζ = eπi/n. This is the unique DSVD of Fn. So the singularvalues are

√n, . . . ,

√n (n times), rank(Fn) = n and ‖Fn‖? = n

√n.


Generalization: Group Algebra Multiplication Tensor

G is a group with n elements and CG ∼= Cn is the group algebra

TG =∑

g ,h∈Gg ⊗ h ⊗ h−1g−1.

DFT case corresponds to G = Z/nZ.

Theorem (D.)

TG has a DSVD and its singular values are√nd1, . . . ,

√nd1︸︷︷︸

d31

, . . . ,√

nds, . . . ,

√nds︸︷︷︸

d3s

where d1, d2, . . . , ds are the dimension of the irreduciblerepresentations of G .


Generalization: Group Algebra Multiplication Tensor

G is a group with n elements and CG ∼= Cn is the group algebra

TG =∑

g ,h∈Gg ⊗ h ⊗ h−1g−1.

DFT case corresponds to G = Z/nZ.

Theorem (D.)

TG has a DSVD and its singular values are√nd1, . . . ,

√nd1︸︷︷︸

d31

, . . . ,√

nds, . . . ,

√nds︸︷︷︸

d3s

where d1, d2, . . . , ds are the dimension of the irreduciblerepresentations of G .


Pn no DSVD

Suppose that Pn has singular values λ1, . . . , λr .

‖Pn‖? = nn/2 =∑r

i=1 λi

‖Pn‖σ = n!nn/2

= λ1

‖Pn‖22 = n! =∑r

i=1 λ2i

λ1

r∑i=1

λi = n! =r∑

i=1

λ2i

so λ1 = · · · = λr , and r = rλ1λ1

= ‖Dn‖?‖Dn‖σ = nn

n! .If n > 2 then r is not an integer!


Dn no DSVD

Suppose that Dn has singular values λ1, . . . , λr . Thenλ1 = · · · = λr = 1 and r = n! (similar calculation).

v1, v2, . . . , vr

are 2-orthogonal, so r ≤ (dimV )1/2 = nn/2. So

n! = r ≤ nn/2

Not possible for n > 2.


Slope decomposition

Definition

T = T1 + T2 + · · ·+ Ts is called a slope decomposition if〈Ti ,Tj〉 = ‖Ti‖?‖Tj‖σ for all i ≤ j and

‖T1‖?‖T1‖σ

<‖T2‖?‖T2‖σ

< · · · < ‖Tr‖?‖Tr‖σ

.

Slope decomposition is unique if it exists. If T has a DSVD, thenit also has a slope decomposition. But not every tensor has a slopedecomposition.


Singular values for tensors with slope decomposition

Definition

Suppose that T = T1 + · · ·+ Ts is a slope decomposition,µi = ‖Ti‖σ, and λi = µi + µi+1 + · · ·+ µs for all i . Then T has

singular value λi with multiplicity ‖Ti‖?‖Ti‖σ −

‖Ti−1‖?‖Ti−1‖σ .

Dn has singular value 1 with multiplicity n!Pn has singular value n!

nn/2with multiplicity nn

n! .

Lemma

If T has slope decomposition and singular values λ1, . . . , λr withmultiplicities m1, . . . ,mr , then ‖T‖σ = λ1, ‖T‖? =

∑i miλi and

‖T‖22 =∑

i miλ2i .



Definition



‖Ti−1‖?‖Ti−1‖σ .



n! .

Lemma


∑i miλi and

‖T‖22 =∑

i miλ2i .



Definition



‖Ti−1‖?‖Ti−1‖σ .



n! .

Lemma


∑i miλi and

‖T‖22 =∑

i miλ2i .


Generalizations

One can a singular spectrum for arbitrary tensors compatible withthe singular spectrum for tensors with slope decomposition, but:one may get continuous spectra or negative multiplicities.

One can define a slope decomposition and singular values wheneverwe have a vector space with two norms dual to each other.

Thank You!


Generalizations



Thank You!


Generalizations



Thank You!


Documents

Tensor Decompositions, Matrix Completion and Singular Valuessites.lsa.umich.edu/hderksen/wp-content/uploads/sites/614/2018/09/… · Tensor Decompositions, Matrix Completion and Singular