EECS 275 Matrix Computation

Ming-Hsuan Yang

Electrical Engineering and Computer ScienceUniversity of California at Merced

Merced, CA 95344http://faculty.ucmerced.edu/mhyang

Lecture 12

1 / 18

Overview

QR decomposition by Householder transformation

QR decomposition by Givens rotation

2 / 18

Reading

Chapter 10 of Numerical Linear Algebra by Llyod Trefethen andDavid Bau

Chapter 5 of Matrix Computations by Gene Golub and Charles VanLoan

Chapter 5 of Matrix Analysis and Applied Linear Algebra by CarlMeyer

3 / 18

Householder triangularization

In Gram-Schmidt process

AR1R2 · · ·Rn︸ ︷︷ ︸R−1

= Q

has orthonormal columns. The product R = R−1n · · ·R−12 R−11 is uppertriangular

In Householder triangularization, a series of elementary orthogonalmatrices Qk is applied to the left of A

Qn · · ·Q2Q1︸ ︷︷ ︸Q>

A = R

is upper triangular. The product Q = Q>1 Q>2 · · ·Q>n is orthogonal,and therefore A = QR is a QR factorization of A

4 / 18

Geometry of elementary projectors

For u, x ∈ IRn, s.t. ‖u‖ = 1

Orthogonal projectors onto span{u} and u⊥ are

Pu =uu>

u>u, and Pu⊥ = I − uu>

u>u

For u 6= 0, the Householder transformation or the elementary reflectorabout u⊥ is

R = I − 2uu>

u>uor

R = I − 2uu>

when ‖u‖ = 1, andR = R> = R−1

5 / 18

Triangularization by introducing zeros

The matrix Qk is chosen to introduce zeros below the diagonal in thek-th column while preserving all the zeros previously introduced× × ×× × ×× × ×× × ×× × ×

Q1−→

� � �0 � �0 � �0 � �0 � �

Q2−→

× × ×

� �0 �0 �0 �

Q3−→

× × ×× ×

�00

A Q1A Q2Q1A Q3Q2Q1A

Qk operates on row k, . . . ,m (changed entries are denoted byboldface or � and blank entries are zero)

At beginning of step k , there is a block of zeros in the first k − 1columns of these rows

The application of Qk forms linear combinations of these rows, andthe linear combination of the zero entries remain zero

After n steps, all the entries below the diagonal have been eliminatedand Qn · · ·Q2Q1A = R is upper triangular

6 / 18

Householder reflectors

At beginning of step k , there is a block of zeros in the first k − 1columns of these rows

Each Qk is chosen to be

Qk =

[I 00 F

]where I is the (k − 1)× (k − 1) identity matrix and F is an(m − k + 1)× (m − k + 1) orthogonal matrix

Multiplication by F has to introduce zeros into the k-th column

The Householder algorithm chooses F to be a particular matrix calledHouseholder reflector

At step k , the entries k, . . . ,m of the k-th column are given by vectorx ∈ IRm−k+1

7 / 18

Householder transformation (cont’d)

To introduce zeros into k-th column (x ∈ IRm−k+1), the Householdertransformation F should

x =

××...×

F−→ Fx =

‖x‖

0...0

= ‖x‖e1 = αe1

The reflector F will reflect the space IRm−k+1 across the hyperplaneH orthogonal to u = ‖x‖e1 − x

A hyperplane is characterized by a vector u = ‖x‖e1 − x

8 / 18

Householder transformation (cont’d)

Every point x ∈ IRm is mapped to a mirror point

Fx = (I − 2uu>

u>u)x = x− 2u(

u>x

u>u)

and hence

F = (I − 2uu>

u>u)

Will fix the +/- sign in the next slide

9 / 18

The better of two Householder reflectors

Two Householder reflectors (transformations)

For numerical stability pick the one that moves reflect x to the vector‖x‖e1 that is not to close to x itself, i.e., −‖x‖e1 − x in this case

In other words, the better of the two reflectors is

u = sign(x1)‖x‖e1 + x

where x1 is the first element of x (sign(x1) = 1 if x1 = 0)

10 / 18

Householder QR factorization

Algorithm:

for k = 1 to n dox = Ak:m,k

uk = sign(x1)‖x‖2e1 + xuk = uk

‖uk‖2Ak:m,k:n = (I − 2uku

>k )Ak:m,k:n

end for

Recall

Qk =

[I 00 F

]Upon completion, A has been reduced to upper triangular form, i.e.,R in A = QR

Q> = Qn · · ·Q2Q1 or Q = Q>1 Q>2 · · ·Q>n

11 / 18

QR decomposition with Householder transformation

Want to compute QR decomposition A with Householdertransformation

A =

12 −51 46 167 −68−4 24 −41

Need to find a reflector for first column of A, x = [12, 6,−4]> to‖x‖e1 = [14, 0, 0]>

u=‖x‖e1 − x = [2,−6, 4]> = 2[1,−3, 2]>

F1=I − 2uu>

u>u=

6/7 3/7 −2/73/7 −2/7 6/7−2/7 6/7 3/7

,F1A =

14 21 −140 −49 −140 168 −77

Next need to zero out A32 and apply the same process to

A′ =

[−49 −14168 −77

]12 / 18

QR decomposition with Householder (cont’d)

With the same process

F2 =

1 0 00 −7/25 24/250 24/25 7/25

Thus, we have

Q = Q1Q2 =

6/7 −69/175 58/1753/7 158/175 −6/175−2/7 6/35 33/35

R = Q2Q1A = Q>A =

14 21 −140 175 −700 0 −35

The matrix Q is orthogonal and R is upper triangular

13 / 18

Givens rotationsGivens rotation: orthogonal transform to zero out elements selectively

G (i , k , θ) =

1 · · · 0 · · · 0 · · · 0...

. . ....

......

0 · · · c · · · s · · · 0...

... · · ·...

...0 · · · −s · · · c · · · 0...

......

. . ....

0 · · · 0 · · · 0 · · · 1

i

k

i kwhere c = cos(θ) and s = sin(θ) for some θPre-multiply G (i , k , θ) amounts to a counterclockwise rotation θ inthe (i , k) coordinate plane, y = G (i , k , θ)x

yj =

cxi − sxk j = i

sxi + cxk j = k

xj j 6= i , k14 / 18

Givens rotations (cont’d)

Can zero out yk = sxi + cxk = 0 by setting

c =xi√

x2i + x2k

, s =−xk√x2i + x2k

, θ = arctan(xk/xi )

QR decomposition can be computed by a series of Givens rotations

Each rotation zeros an element in the subdiagonal of the matrix,forming R matrix, Q = G1 . . .Gn forms the orthogonal Q matrix

Useful for zero out few elements off diagonal (e.g., sparse matrix)

Example

A =

12 −51 46 167 −68−4 24 −41

Want to zero out A31 = −4 with rotation vector (6,−4) to pointalong the x-axis, i.e., θ = arctan(−4/6)

15 / 18

QR factorization with Givens rotation

With θ we have the orthogonal Givens rotation G1

G1 =

1 0 00 cos(θ) sin(θ)0 −sin(θ) cos(θ)

=

1 0 00 0.83205 −0.554700 0.55470 0.83205

Pre-multiply A with G1

G1A =

12 −51 47.2110 125.6396 −33.83671

0 112.6041 −71.83368

Continue to zero out A21 and A32 and form a triangular matrix R

The orthogonal matrix Q> = G3G2G1, and G3G2G1A = Q>A = R forQR decomposition

16 / 18

Gram-Schmidt, Householder and Givens

Householder QR is numerically more stable

Gram-Schmidt computes orthonormal basis incrementally

Givens rotation is more useful for zero out few selective elements

17 / 18

Eigendecomposition

Also known as spectral decomposition

A is a square matrixA = QDQ−1

where Q is a square matrix whose columns are eigenvector and D is adiagonal matrix whose elements are the corresponding eigenvalues

With eigendecomposition

AQ = QDAqi = λiqi

where λi and qi are eigenvalues and eigenvectors of

Ax = λx

The eigenvectors are usually normalized but not necessarily

If A can be eigendecomposed with all non-zero eigenvalues

A−1 = QD−1Q−1

18 / 18