The role of tensor rank in the complexity analysis of ...lekheng/meetings/multilinear/bini.pdf · Dario A. Bini The role of tensor rank. Preliminaries Properties of tensor rank Open

PreliminariesProperties of tensor rank

Open problems

The role of tensor rank in the complexity analysisof bilinear forms

Dario A. Bini

Dipartimento di Matematica, Universita di Pisawww.dm.unipi.it/˜bini

ICIAM07, Zurich, 16-20 July 2007

Dario A. Bini The role of tensor rank


Open problems

1 PreliminariesTensorsTensor rankBilinear forms

2 Properties of tensor rankBorder rankLower bounds

3 Open problemsThe direct sum conjectureSome tensors of unknown rank



Open problems

TensorsTensor rankBilinear forms

Tensors

Let F be a number field, say, R, C; tensors of the kind

A = (ai1,i2,...,ih) ∈ Fn1×n2×···×nh ,

that is, h-way arrays, are encountered in many problems of verydifferent nature [Comon 2001], [Comon, Golub, Lim, Mourrain,2006], [De Silva, Lim 2006]

Blind source separation

High order factor analysis

Independent component analysis

Candecomp/Parafac model

Complexity analysis

Psycometric, Chemometric, Economy,...



Open problems


Tensors

It is surprising that little interplay occurred among thesedifferent research areas

Some properties have been rediscovered in different contexts

Apparently, results obtained in one field have not migrated tothe other fields

In this talk I wish to provide an overview of the main resultsconcerning tensors obtained in the research field of computationalcomplexity (starting from 1969) with the aim of

creating a synergic exchange of information between theseresearch areas

presenting problems which might be solved with the morerecent tools

presenting old results that might be adapted and extended tothe new needs



Open problems


Tensors

It is surprising that little interplay occurred among thesedifferent research areas

Some properties have been rediscovered in different contexts

Apparently, results obtained in one field have not migrated tothe other fields

In this talk I wish to provide an overview of the main resultsconcerning tensors obtained in the research field of computationalcomplexity (starting from 1969) with the aim of

creating a synergic exchange of information between theseresearch areas

presenting problems which might be solved with the morerecent tools

presenting old results that might be adapted and extended tothe new needs



Open problems


Tensor rank

Definition (Hitchcock 1927)

A tensor T = (ti1,...,ih) has rank 1 if there exist vectors

u(k) = (u(k)i ) ∈ Fnk , k = 1 : h such that ti1,...,ih = u

(1)i1

u(2)i2· · · u(h)

ih,

T = u(1) ◦ u(2) ◦ · · · ◦ u(h)

Definition (Hitchcock 1927)

The tensor rank rk(A) of A = (ai1,...,ih) is the minimum number rof rank-1 tensors Ti ∈ Fn1×···×nh such that

A = T1 + T2 + · · ·+ Tr canonical decomposition

For matrices, the tensor rank coincides with the customary rank

For simplicity of notation we restrict ourselves to the case h = 3,i.e., to three-way arrays.



Open problems


Some remarks

For A ∈ Fm×n×p a canonical decomposition

A = u(1) ◦ v(1) ◦w(1) + u(2) ◦ v(2) ◦w(2) + · · ·+ u(r) ◦ v(r) ◦w(r)

is defined by three matrices

U = (ui ,j) ∈ Fm×r , V = (vi ,j) ∈ Fn×r , W = (wi ,j) ∈ Fp×r ,

whose columns are the vectors u(i), v(i), w(i), i = 1 : r ,respectively.



Open problems


Some remarks

A tensor A = (ai ,j ,k) ∈ Fm×n×p, can be represented by means ofthe set of the 3-slabs Ak = (ai ,j ,k)i ,j ∈ Fm×n of A or by a singlematrix of variables

A =

p∑k=1

skAk

A =

[[1 00 −1

];

[0 11 0

]]↔

[s1 s2s2 −s1

]

This suggests a different point of view for tensor rank:

rk(A) is the minimum set of rank one matrices which span the linearspace generated by the 3-slabs A1, . . . ,Ap [Gastinel 71, Fiduccia 72].



Open problems


Some remarks

A tensor A = (ai ,j ,k) ∈ Fm×n×p, can be represented by means ofthe set of the 3-slabs Ak = (ai ,j ,k)i ,j ∈ Fm×n of A or by a singlematrix of variables

A =

p∑k=1

skAk

A =

[[1 00 −1

];

[0 11 0

]]↔

[s1 s2s2 −s1

]This suggests a different point of view for tensor rank:

rk(A) is the minimum set of rank one matrices which span the linearspace generated by the 3-slabs A1, . . . ,Ap [Gastinel 71, Fiduccia 72].



Open problems


Bilinear forms

Problem: Given matrices Ak = (ai ,j ,k)i=1:m,j=1:n, k = 1 : pcompute the set of bilinear forms

fk(x, y) = xTAky, k = 1 : p

with the minimum number of nonscalar multiplications with no useof commutativity (noncommutative bilinear complexity)

A nonscalar multiplication is a multiplication of the kind

s = (m∑

i=1

αixi )(n∑

j=1

βjyj), αi , βj ∈ F

Remark: The set of bilinear forms is uniquely determined by thetensor A = (ai ,j ,k).



Open problems


Bilinear forms

A canonical decomposition of the tensor A associated with the setof bilinear forms provides an algorithm of complexity r . In fact

A =r∑

`=1

u(`) ◦ v(`) ◦w(`) → Ak =r∑

`=1

wk,`u(`) ◦ v(`)

→ xTAky =r∑

`=1

wk,`(xTu(`))(v(`)Ty)

Theorem (Strassen 1975)

The noncommutative bilinear complexity of a set of bilinear formsfk(x, y) = xTAky, k = 1 : p, Ak = (ai ,j ,k), is given by the tensorrank of the associated tensor A = (ai ,j ,k).



Open problems


An example

Multiplication of complex numbers

(x1 + ix2)(y1 + iy2) = (x1y1 − x2y2) + i(x1y2 + x2y1) = f1 + if2

Apparently, 4 multiplications are needed

Tensor: A =

[[1 00 −1

];

[0 11 0

]]The tensor rank is at most 3:[

1 00 −1

]=

[1 00 0

]−

[0 00 1

][

0 11 0

]=

[1 11 1

]−

[1 00 0

]−

[0 00 1

]Algorithm:

s1 = (x1 + x2)(y1 + y2),s2 = x1y1,s3 = x2y2

f1 = s2 − s3,f2 = s1 − s2 − s3



Open problems


An example




Tensor: A =

[[1 00 −1

];

[0 11 0

]]

The tensor rank is at most 3:[1 00 −1

]=

[1 00 0

]−

[0 00 1

][

0 11 0

]=

[1 11 1

]−

[1 00 0

]−

[0 00 1

]Algorithm:

s1 = (x1 + x2)(y1 + y2),s2 = x1y1,s3 = x2y2

f1 = s2 − s3,f2 = s1 − s2 − s3



Open problems


An example




Tensor: A =

[[1 00 −1

];

[0 11 0

]]The tensor rank is at most 3:[

1 00 −1

]=

[1 00 0

]−

[0 00 1

][

0 11 0

]=

[1 11 1

]−

[1 00 0

]−

[0 00 1

]Algorithm:

s1 = (x1 + x2)(y1 + y2),s2 = x1y1,s3 = x2y2

f1 = s2 − s3,f2 = s1 − s2 − s3



Open problems

Border rankLower bounds

Tensor rank

Main problem: For a given tensor, compute its rank and a canonicaldecomposition.

Two ways of attacking the problem

looking for lower bounds to the tensor ranklooking for upper bounds to the tensor rank

Things get more complicate: unlike the case of m × n matrices

The rank depends on the ground field FHigh rank tensors can be approximated by low rank tensors

Rational algorithms for tensor rank, like Gaussian elimination, can-not exist



Open problems


Topological properties of tensors: Border Rank

For m × n matrices, any sequence {Ak} of m × n matrices of rankr with limit A = limk Ak is such that rank(A) ≤ r .

For higher order tensor this property does not hold anymore

Example (Bini, Capovani, Lotti, Romani 1980)

The tensor

A =

[[1 00 1

];

[0 10 0

]]has rank 3. The tensor

Aε =

[[1 0ε 1

];

[0 10 0

]]has rank 2 for any ε 6= 0



Open problems


Topological properties of tensors: Border Rank

For m × n matrices, any sequence {Ak} of m × n matrices of rankr with limit A = limk Ak is such that rank(A) ≤ r .

For higher order tensor this property does not hold anymore

Example (Bini, Capovani, Lotti, Romani 1980)

The tensor

A =

[[1 00 1

];

[0 10 0

]]has rank 3. The tensor

Aε =

[[1 0ε 1

];

[0 10 0

]]has rank 2 for any ε 6= 0



Open problems


Canonical decomposition

[1 0ε 1

]=

[1 ε−1

ε 1

]− ε−1

[0 10 0

][

0 10 0

]=

[0 10 0

]



Open problems


Border rank

Definition (Bini, Capovani, Lotti, Romani 1980)

The border rank of A ∈ Fm×n×p (F = R, C) is

brk(A) = min{r : ∀ε > 0 ∃E ∈ Fm×n×p : ||E|| < ε, rk(A+E) = r}

where || · || is any norm

Some properties:

brk(A) ≤ rk(A)

brk(A) is the minimum number of nonscalar multiplicationssufficient to approximate the set of bilinear forms associatedwith A with arbitrarily small nonzero error



Open problems


More on rank and border rank

Let Aε = A+ Eε be such that

rk(Aε) = brk(A)the entries of Eε are polynomials of degree d

Then

rk(A) ≤ (d + 1)brk(A)

Proof:

Write d + 1 copies of the canonical decomposition of lengthrk(Aε) obtained with (d + 1) pairwise different values of εTake linear combinations of these decompositions withcoefficients γj , j = 1 : d + 1 in order to kill the terms in εi ,i = 1 : d and to have

∑γj = 1

Obtain a decomposition of length (d + 1)brk(A) with no error



Open problems





Then

rk(A) ≤ (d + 1)brk(A)

Proof:

Write d + 1 copies of the canonical decomposition of lengthrk(Aε) obtained with (d + 1) pairwise different values of ε

Take linear combinations of these decompositions withcoefficients γj , j = 1 : d + 1 in order to kill the terms in εi ,i = 1 : d and to have

∑γj = 1




Open problems





Then

rk(A) ≤ (d + 1)brk(A)

Proof:


∑γj = 1




Open problems





Then

rk(A) ≤ (d + 1)brk(A)

Proof:


∑γj = 1




Open problems


Lower bounds and linear algebra

Simple criteria for providing lower bounds on tensor rank andborder rank can be givenTrivial bounds

brk(A) ≥ dim(span(A1, . . . ,Ap))

Similar inequalities are valid w.r.t. the other coordinates

Assume for simplicity p = dim(span(A1, . . . ,Ap))



Open problems



Let U,V ,W be the matrices defining a canonical factorization ofA of length rk(A)

A =

rk(A)∑`=1

u(`) ◦ v(`) ◦w(`)

Assume w.l.o.g. that the first p columns of W are linearlyindependent, that is

W =[

W1 W2

], W1 ∈ Fp×p, det W1 6= 0

Let A′k =

∑pj=1 w

(−1)k,j Aj , define

A′ = [A′1,A

′2, . . . ,A

′p] = A •3 W−1

1 ,

i.e., choose a different basis to represent the space spanned by theslabs A1, . . . ,Ap



Open problems




A =

rk(A)∑`=1

u(`) ◦ v(`) ◦w(`)


W =[

W1 W2

], W1 ∈ Fp×p, det W1 6= 0

Let A′k =

∑pj=1 w


A′ = [A′1,A

′2, . . . ,A

′p] = A •3 W−1

1 ,




Open problems




A =

rk(A)∑`=1

u(`) ◦ v(`) ◦w(`)


W =[

W1 W2

], W1 ∈ Fp×p, det W1 6= 0

Let A′k =

∑pj=1 w


A′ = [A′1,A

′2, . . . ,A

′p] = A •3 W−1

1 ,




Open problems


Evidently, A′ has the same rank of A and a canonicaldecomposition is given by U ′ = U,V ′ = V ,W ′ = W−1

1 W , where

W ′ =[

I W−11 W2

]=

1 0 0 0 ∗ ∗ ∗0 1 0 0 ∗ ∗ ∗0 0 1 0 ∗ ∗ ∗0 0 0 1 ∗ ∗ ∗

Remark

Any subtensor A formed by k 3-slabs [A′σ1

, . . . ,A′σk

] is such that

rk(A) ≤ rk(A)− (p − k)



Open problems


More generally,

Theorem

rk(A) ≥ maxT

minS

(p − k + rk(A •3 TS))

T: k × p submatrix of Ip, S: p × p nonsingular

Corollary

If for any basis of span(A1, . . . ,Ap) there exists a matrix of rank qthen

rk(A) ≥ p + q − 1



Open problems


More generally,

Theorem

rk(A) ≥ maxT

minS

(p − k + rk(A •3 TS))


Corollary

If for any basis of span(A1, . . . ,Ap) there exists a matrix of rank qthen

rk(A) ≥ p + q − 1



Open problems


Higher order generalization:

Corollary

Let A = [A1, . . . ,Anh] ∈ Fn1×···×nh , Ak ∈ Fn1×···×nh−1 be such

that nh = dim(span(A1, . . . ,Anh)). If for any basis of

span(A1, . . . ,Anh) there exists a tensor in the basis of rank q then

rk(A) ≥ nh + q − 1

Example: For

A =

[[1 00 1

];

[0 10 0

]]any basis must contain a nonsingular matrix, therefore

rk(A) ≥ 2 + 2− 1 = 3



Open problems


Higher order generalization:

Corollary

Let A = [A1, . . . ,Anh] ∈ Fn1×···×nh , Ak ∈ Fn1×···×nh−1 be such

that nh = dim(span(A1, . . . ,Anh)). If for any basis of

span(A1, . . . ,Anh) there exists a tensor in the basis of rank q then

rk(A) ≥ nh + q − 1

Example: For

A =

[[1 00 1

];

[0 10 0

]]any basis must contain a nonsingular matrix, therefore

rk(A) ≥ 2 + 2− 1 = 3



Open problems


Lower bounds to border rank

Unfortunately the same technique cannot be applied to border rank

W1 may be singular in the limit as ε → 0 so that E •3 W−11 may not be

infinitesimal anymore

Remedy: compute the QR factorization of W and multiply W by QT

which has Euclidean norm 1 for any ε. One has

W ′ = QTW =

∗ ∗ ∗ ∗ ∗ ∗ ∗0 ∗ ∗ ∗ ∗ ∗ ∗0 0 ∗ ∗ ∗ ∗ ∗0 0 0 ∗ ∗ ∗ ∗

Remark

There exists an m × n × k subtensor A such that

brk(A) ≤ brk(A)− (p − k)



Open problems


Theorem

brk(A) ≥ minT

minS

(p − k + brk(A •3 TS))


Corollary

If any basis of the linear space spanned by A1, . . . ,Ap is made upby matrices of rank at most q then brk(A) ≥ p + q − 1

Corollary (Generalization to higher dimension)

Let A = [A1, . . . ,Anh]. If any basis of the linear space spanned by

A1, . . . ,Anhis made up by tensors of rank at most q then

brk(A) ≥ nh + q − 1



Open problems


Theorem

brk(A) ≥ minT

minS

(p − k + brk(A •3 TS))


Corollary





brk(A) ≥ nh + q − 1



Open problems


Theorem

brk(A) ≥ minT

minS

(p − k + brk(A •3 TS))


Corollary





brk(A) ≥ nh + q − 1



Open problems


Example

The tensor A ∈ Fn2×n2×n2whose 3-slabs span the linear space

I ⊗ S =

S. . .

S

, S =

s1,1 . . . s1,n...

. . ....

sn,1 . . . sn,n

is associated with n × n matrix multiplication.

All the matrices in any basis of the linear space have rank at leastn. Therefore

brk(A) ≥ n2 + n − 1

The bound can be improved to

brk(A) ≥ n2 + 2n − 2



Open problems


Example

The tensor A ∈ Fn2×n2×n2whose 3-slabs span the linear space

I ⊗ S =

S. . .

S

, S =

s1,1 . . . s1,n...

. . ....

sn,1 . . . sn,n

is associated with n × n matrix multiplication.

All the matrices in any basis of the linear space have rank at leastn. Therefore

brk(A) ≥ n2 + n − 1

The bound can be improved to

brk(A) ≥ n2 + 2n − 2



Open problems


Example

The tensor A ∈ F4×4×3 whose 3-slabs span the linear space s1 s2 0 0s3 s4 0 0

0 0 s1 s2

is associated with the multiplication of a 2× 2 triangular matrixand a full 2× 2 matrix

For any basis of the linear space there exist two matrices whichform a tensor of rank at least 4.

rk(A) ≥ 4− 2 + 4 = 6

Since there exists a canonical approximate decomposition of length5 of A then brk(A) ≤ 5 < 6 = rk(A)



Open problems


Example

The tensor A ∈ F4×4×3 whose 3-slabs span the linear space s1 s2 0 0s3 s4 0 0

0 0 s1 s2

is associated with the multiplication of a 2× 2 triangular matrixand a full 2× 2 matrix

For any basis of the linear space there exist two matrices whichform a tensor of rank at least 4.

rk(A) ≥ 4− 2 + 4 = 6

Since there exists a canonical approximate decomposition of length5 of A then brk(A) ≤ 5 < 6 = rk(A)



Open problems

The direct sum conjectureSome tensors of unknown rank

The direct sum conjecture

Let A ∈ Fm×n×p, B ∈ Fm′×n′×p′ . Consider the direct sum of Aand B

C = A⊕ B ∈ F(m+m′)×(n+n′)×(p+p′)

Direct sum conjecture, Strassen 1973

rk(C) = rk(A) + rk(B)

The complexity of two disjoint sets of bilinear forms is the sum ofthe complexities of each set.

Fact

There are cases where brk(C) < brk(A) + brk(B)



Open problems


The direct sum conjecture

Let A ∈ Fm×n×p, B ∈ Fm′×n′×p′ . Consider the direct sum of Aand B

C = A⊕ B ∈ F(m+m′)×(n+n′)×(p+p′)

Direct sum conjecture, Strassen 1973

rk(C) = rk(A) + rk(B)

The complexity of two disjoint sets of bilinear forms is the sum ofthe complexities of each set.

Fact

There are cases where brk(C) < brk(A) + brk(B)



Open problems


Example (A. Schoenhage 81)

brk(A) = 10 < 9 + 4

A =

s11 s12 s13s21 s22 s23s21 s22 s23

tt

tt



Open problems


. . .

. 1 1 ε ε

. 1 1 ε ε

ε ε2

ε ε2

ε ε2

ε ε2

1 1 11 . . −ε −ε1 . . −ε −ε

−ε−ε

−ε−ε



Open problems


Matrix multiplication and open problems

2× 2 matrix product:

A = I2 ⊗[

s11 s12s21 s22

]=

s11 s12s21 s22

s11 s12

s21 s22

rk(A) = brk(A) = 7 [Strassen 69, Landsberg 05]

3× 3 matrix product A = I3 ⊗

s11 s12 s13

s21 s22 s23s31 s32 s33

19 ≤ rk(A) ≤ 23 [Laderman 76], [Blaser 03]

13 ≤ brk(A) ≤ 22 [Schonhage 81]



Open problems


Matrix multiplication and open problems

4× 4 matrix product A = I4 ⊗

s11 s12 s13 s14s21 s22 s23 s24s31 s32 s33 s34s41 s42 s43 s44

34 ≤ rk(A) ≤ 49

22 ≤ brk(A) ≤ 49


Documents

The role of tensor rank in the complexity analysis of ...lekheng/meetings/multilinear/bini.pdf · Dario A. Bini The role of tensor rank. Preliminaries Properties of tensor rank Open