Scientific Computing
Singular Value DecompositionSVD
SVD - Overview
SVD is a way to decompose singular (or nearly singular) matrices, i.e. matrices that do not have inverses. This includes square matrices whose determinant is zero (or nearly zero) and all rectangular matrices.
SVD - Basics
The SVD of a m-by-n matrix A is given by the formula :
TUDVAWhere :U is a m-by-m matrix of the orthonormal eigenvectors of AAT (U is orthogonal)VT is the transpose of a n-by-n matrix containing the orthonormal eigenvectors of ATA (V is orthogonal)D is a n-by-n Diagonal matrix of the singular values which are the square roots of the eigenvalues of ATA
SVD
• In Matlab: [U,D,V]=svd(A,0)
T1
00
00
00
VUA
nw
w
T1
00
00
00
VUA
nw
w
The Algorithm
Derivation of the SVD can be broken down into two major steps [2] :1. Reduce the initial matrix to bidiagonal form using
Householder transformations (reflections)2. Diagonalize the resulting matrix using orthogonal
transformations (rotations)
Initial Matrix Bidiagonal Form Diagonal Form
Householder Transformations
Recall: A Householder matrix is a reflection defined as :H = I – 2wwT
Where w is a unit vector with |w|2 = 1.We have the following properties :
H = HT
H-1 = HT
H2 = I (Identity Matrix)
If H is multiplied by another matrix, (on right/left) it results in a new matrix with zero’ed out elements in a selected row / column based on the values chosen for w.
Applying HouseholderTo derive the bidiagonal matrix, we apply successive
Householder matrices on the left (columns) and right (rows):
21 M
21
MM
1
xxxx
xxxx
xxxx
xxxx
xx
HK
xxxx
xxxx
xxxx
xxxx
xxxxx
xxxxx
xxxxx
xxxxx
xxxxx
xxxxx
H
m
x
x
xxx
xxxx
xx
HK
xxx
xxx
xxx
xxxx
xx
m
M
2
M
......
3
BMn
x
xx
xx
xx
xx
K
x
xx
xxx
xx
xx
n
Application con’t
Householder CalculationColumns: Recall that we zero out the column below the (k,k) entry
as follows (note that there are m rows): Let (column vector-size m)
Note:
Thus, where Ik is a kxk identity matrix.
),,,,0,,0()sgn(
),,,,0,,0(
,,1,,
,,1,
kkm
kkk
kkk
kkkk
kkm
kkkk
kkkk
aaaa
aaav
tkkk
k
kk wwIHand
v
vw 2
2
**00
**00
0000
**0
*
*
0
tkkww
M
IwwIH tkkkk 0
0)2(
Householder CalculationRows: To zero out the row past the (k,k+1) entry: Let (row vector- size n)
where wk is a row vector
Note:
Thus, where Ik is a (k-1)x(k-1) identity matrix.
),,,,0,,0()sgn(
),,,0,,0(
,2,1,1,
,1,
knk
kkk
kkk
kkkk
knkk
kkkk
aaaa
aav
ktkk
k
kk wwIHand
v
vw 2
2
**00
**00
0000
**0
*
*
0
k
tkww
€
Kk = (Ik − 2wktwk ) =
I 0
0 M
⎡
⎣ ⎢
⎤
⎦ ⎥
ExampleTo derive H1 for the given matrix A :
We have :
Thus,
So,
4
2
4
1A
4
2
10
0
0
1
6
4
2
4
6
4
2
4
1 11 v
ttt
w
30
2,
30
1,
30
5
120
4,
120
2,
120
10
4210
4,2,10
2221
15
11
15
2
3
215
2
15
14
3
13
2
3
1
3
2
15
2
15
1
3
115
1
30
1
6
13
1
6
1
6
5
230
2,
30
1,
30
5
30
230
130
5
221 IIwwIH T
3044
1212
2034
Example con’tThen,
For K1:
733.0267.08.00
133.0867.16.00
667.3667.056
15/1115/45/40
15/215/285/30
3/113/256
3044
1212
2034
15
11
15
2
3
215
2
15
14
3
13
2
3
1
3
2
1AH
667.3667.0236.1100010236.6667.3667.050
236.6667.3667.0505
2
1
v
3098.00563.0949.002
22
v
vw
8081.00349.0588.00
0349.09937.0107.00
588.0107.08017.00
0001
096.00175.0294.00
0175.00032.00535.00
294.00535.0901.00
0000
2
3098.00563.0949.00
3098.0
0563.0
949.0
0
22 111
I
IwwIK t
Example con’tThen,
We can start to see the bidiagonal form.
€
H1AK1 =
−6 −5 −0.667 −3.667
0 −0.6 1.867 −0.133
0 0.8 −0.267 0.733
⎡
⎣
⎢ ⎢ ⎢
⎤
⎦
⎥ ⎥ ⎥
1 0 0 0
0 −0.8017 −0.107 −0.588
0 −0.107 0.9937 −0.0349
0 −0.588 −0.0349 0.8081
⎡
⎣
⎢ ⎢ ⎢ ⎢
⎤
⎦
⎥ ⎥ ⎥ ⎥
=
−6 6.236 0 0
0 0.3598 1.9236 0.1799
0 −1.0441 −0.3761 0.1315
⎡
⎣
⎢ ⎢ ⎢
⎤
⎦
⎥ ⎥ ⎥
Example con’tIf we carry out this one more time we get:
€
B = HAK
B =
−6 6.236 0 0
0 −1.1044 0.9846 0
0 0 −1.678 0.3257
⎡
⎣
⎢ ⎢ ⎢
⎤
⎦
⎥ ⎥ ⎥
H =
−0.667 −0.522 −0.532
−0.333 −0.430 0.839
−0.667 0.737 0.113
⎡
⎣
⎢ ⎢ ⎢
⎤
⎦
⎥ ⎥ ⎥
K =
1 0 0 0
0 −0.802 0.0674 −0.594
0 −0.107 −0.994 0.0315
0 −0.588 0.0888 0.804
⎡
⎣
⎢ ⎢ ⎢ ⎢
⎤
⎦
⎥ ⎥ ⎥ ⎥
The QR Algorithm
As seen, the initial matrix is placed into bidiagonal form which results in the following decomposition :
A = HBK with H = H1...Hn and K = Km…K1
The next step takes B and converts it to the final diagonal form using successive rotation transformations (as reflections would disrupt upper triangular form).
Givens Rotations
A Givens rotation is used to rotate a plane about two coordinates axes and can be used to zero elements similar to the householder reflection.
It is represented by a matrix of the form :
Note: The multiplication GA effects only the rows i and j in A.Likewise the multiplication AGt only effects the columns i and j.
1000
00
00
0001
),,(jjij
jiii
cs
scjiG
€
c ij = cos(θ )
sij = sin(θ )
Givens rotation
The zeroing of an element is performed by computing the c and s in the following system.
0
22 bab
a
cs
sc
Where b is the element being zeroed and a is next to b in the preceding column / row.
This results in :22 ba
ac
€
s =−b
a2 +b2
Givens Example
In our previous example, we had used Householder transformations to get a bidiagonal matrix:
We can use rotation matrices to zero out the off-diagonal terms
Matlab: [U,D,V]=svd(A,0)
€
B =
−6 6.236 0 0
0 −1.1044 0.9846 0
0 0 −1.678 0.3257
⎡
⎣
⎢ ⎢ ⎢
⎤
⎦
⎥ ⎥ ⎥
SVD ApplicationsCalculation of inverse of A:
So, for mxn A define (pseudo) inverse to be: V D-1 Ut
€
A−1A = A−1UDV T →I = A−1UDV T[1] : Given
€
V = A−1UDV TV →V = A−1UD
€
A =UDV T
[2] : Multiply by A-1
[3] : Multiply by V[4]* : Multiply by D-1
€
VD−1 = A−1UDD−1 →VD−1 = A−1U
€
VD−1UT = A−1UUT →VD−1UT = A−1[5] : Multiply by UT
[6] : Rearranging
€
A−1 =VD−1UT
SVD Applications con’t
• SVD can tell How close a square matrix A is to be singular.• The ratio of the largest singular value to the smallest singular value
can tell us how close a matrix is to be singular:
• A is singular if c is infinite.• A is ill-conditioned if c is too large (machine dependent).
Condition number
SVD Applications con’tData Fitting Problem
SVD Applications con’tImage processing[U,W,V]=svd(A)NewImg=U(:,1)*W(1,1)*V(:,1)’
SVD Applications con’t
• SVD is used as a method for noise reduction. • Let a matrix A represent the noisy signal:
– compute the SVD, – and then discard small singular values of A.
• It can be shown that the small singular values mainly represent the noise, and thus the rank-k matrix Ak represents a filtered signal with less noise.
Digital Signal Processing (DSP)