1 Matrix Fundamentals

A = (aij)mn

=

a12 a1n... . . . ...am1 amn

=a1...

an

(a1, ,an)1.1 Kroneker Product

Amn B

pq= (aijB)

mpnq

1.2 Hatamand Product

Amn Bmn

= (ajbj)mn

1.3 Range and Nullspace

range(A) = {y|y = Ax, for some x Rn} null(A) = {x|Ax = 0}

Theorem 1.1 range (A) = span a1, . . . ,an

2 Orthogonal Vectors and Matrices

2.1 Inner Product

inner product of vectors : x,y Rn,xTy =mi=1

xiyi

The Euclidean length of xmay be written as ||x||2, ||x||2 =

xTx = (mi=1

x2)12

The cosine of x and y is cos = xTy

||x||2||y||2

2.2 Orthogonal Vectors

if xTy = 0, xand yare orthogonal

2.3 Orthogonal Sets

Two sets X,Y are orthogonal, if for every x X and every y Y,xTy = 0A set of nonzero vectors Sis orthogonal if its complement are pairwise orthogonal. If x,y S,xTy = 0,x 6= y

2.4 Orthonormal

x 6= yxTy = 0,xTx = 1

Theorem 2.1 The vectors in a orthogonal set S are linearly independent

1

2.5 Orthogonal Matrix

Q = Rmm,QTQ = QQT = ImQ = Cmm,QHQ = QQH = Im, (QH is the unitary matrix of Q)

Q1 = QT

|det(Q)| = 1 ||Qx||2 = ||x||2

2.6 Column Orthogonal Matrix

Q1 Rmn, n < m, QT1 Q1 = In, Q1QT1 6= Im(rank(Q) n).However, we can construct the matrix from column orthogonal matrix by adding colunmsQ2 Rm(mn), QT2Q2 = Im1, QT1Q2 = 0, Q = [Q1

nQ2mn

]

3 Rank

column rank = row rank

Proposition 3.1 Let A and B be m n and n p matrices, then rank(A) = rank(AT ) = rank(ATA) = rank(AAT ) rank(AB) min{rank(A), rank(B)} rank(AB) = rankA if B is square matrix of full rank rank(A + B) rank(A) + rank(B)

Theorem 3.1 rank(A,B) = rank(A) + rank(B)

rank

(A BC D

)=rank

(B AD C

)=rank

(C DA B

)=rank

(D CB A

)Theorem 3.2 Rank factorization Let Abe an m n nonull matrix of rank r, then thereexist an m r matrix and an r n matrix T, such that A= BT. Moreover, for any m r matrix Band r n matrix Tsuch that A= BT, rank (B) = rank (T) = r.

4 Inverse

A is n m matrix. rank (A)= m. A= (a1, . . . ,am).{a1, . . . ,am}is a group of bases in Rnspace. For any vector Rn, there is a linear combination of bases.

ei =

0...1i0

n1

I = [e1, . . . , em] = (a1, . . . ,am)

b11 b1m... . . . ...bm1 bmm

= AB

2

B is the right inverse of A.Similarly, I= CA, and C is called the left inverse of A.Since C=CI= CAB= B, the left inverse of a matrix is equivalent to its right inverse, wehave AA1 = A1A = I.

Proposition 4.1 For A Rmm, the following conditions are equivalent: Ahas an inverse A1

rank(A) = m range(A)= Rm

null(A) = {0} 0 is not an eigenvalue of A 0 is not a singular value of A det(A) 6= 0

Proposition 4.2 (The Sherman-Merriam-Woodburry formular) m n, Ais diag-onal and invertible.We have

(ABD1C)1mm

= A1 + A1B (DCA1B)1CA1nn

Proposition 4.3 (Matrix Inverse) A, B, C, D are matrices

(

A 00 D

)1=

(A1 0

0 D1

)

(

In 0V Im

)1=

(In 0V Im

)

(

A 0C D

)1=

(A1 0

D1CA1 D1)

(

A B0 D

)1=

(A1 A1BD1

0 D1

)

(

A BC D

)1=

(A1 + A1BQ1CA1 A1BQ1

Q1CA1 Q1)

Q = DCA1B(Schur Complement)(A BC D

)is invertible iff Aand Q are invertible

3

S=(

Im BD10 In

)P=S

(A BC D

)=

(ABD1C 0

C D)

T=

(Im 0

D1C In

), PT=

(ABD1C 0

0 D

)S

(A BC D

)T=

(ABD1C 0

0 D

)=

(A BC D

)=

(11 1221 22

), 1 = =

(11 1221 22

)111 = 11 12122 21, 111 12 = 12122 , 21111 = 122 21

Probabilistic Graphical Model :

X1...Xn

=X N(0,)Xi qXj |Xrex iff ij = 0, Xi qXj iff ij = 0

5 Trace

Trace of an (mm) matrix A = [aij ] is defined as tr(A) =mi=1

aii

Proposition 5.1 tr(1A + 2B) = 1trA + 2trB

trA = tr(AT ) =mi=1

i

tr(AB) = tr(BA), tr(xyT ) =mi=1

(xiyi)

A = (a1, ,an), vec(A) =

a1...an

.tr(AB) = vec(AT )T vec(B) = vec(BT )T vec(A)

tr(ABCD) = vec(DT )T (CT A)vec(B) = vec(AT )T (DT B)vec(C)= vec(BT )T (AT C)vec(D) = vec(CT )T (BT D)vec(A)

tr(AB) = tr(A)tr(B)

4

6 Determinant

A = [aij ]mm, |A| = det(A)

Proposition 6.1 det(A) =mi=1

i

det(A) = mdetA det(AB) = det(A)det(B) det(AT ) = detA det(A1) = 1detA det(Im + AAT ) = det(In + ATA)

det(

A B0 C

)= det(A)det(C)

det

(A BC 0

)= (1)mdet(B)det(C)

Amm Bmn Cnn Dnn

Anonsingular =(

A BC D

)= det(A)det(DCA1B)

Dnonsingular =(

A BC D

)= det(D)det(ABD1C)(

A abT c

)= det(A)det(c bTA1a) = det(A)(c bTA1a)

det(AB) = (detA)n(detB)m

det(A + BC) = det(A)det(I + A1BC) = detAdet(I + CA1B)proof: det(A+BC) = det(A(I+A1BC)) = det(A)det(I+A1BC) = detAdet(I+CA1B)

5