Multivariate Statistics

Matrix Algebra II

W. M. van der VeldUniversity of Amsterdam

Overview

• The determinant of a matrix• The matrix inverse• System of equations

The determinant of a matrix

• The determinant of a matrix is a scalar and is denoted as |A| or det(A). Det(A) only exists when A is a square matrix.

• It has very important mathematical properties, but it is very difficult to provide a substantive definition.

• The determinant is necessary to compute the inverse of a matrix (A-1).– The inverse of a matrix is needed for solving systems of linear

equations; multivariate statistics often comes down to this. – When the determinant is zero, there exists no solution to a

system of linear equations.

• Let’s see how the value of the determinant is found.

The determinant of a matrix

• How to do it? The most simple case, a 2 by 2 matrix .• Det(A)=|A|=?

40

31A

40*34*140

31)( AADet

Cofactors

The determinant of a matrix

• One step further, a 3 by 3 matrix. • Det(A)=|A|=?

)0341(1

40

31*1A

12)160()4840()04(

08

15*2

48

35*2

)8345(2 )8105(2

408

315

221

A

Cofactor

The determinant of a matrix

• You should have noted that for matrices larger than first order, computation of the determinant is a recursive process. This process stops each time a 1 by 1 determinant is encountered, and involves multiplication by the cofactors.

The determinant of a matrix

• Let A be a matrix of order n x n. If we omit one or more rows or columns from A, we obtain a matrix of smaller order, called a minor of the matrix.

• Similarly, we have minors of a determinant, and in particular, if we omit from the determinant the ith row and the jth column, the resulting minor will be square and its determinant will be symbolized |Mij|. This determinant is called a cofactor (cij) if we give it a sign equal to (-1)i+j, so that: cij = (-1)i+j |Mij|. Using this notation we can write a formula for the expansion of a determinant of order n:

In this version the determinant is expanded according to it’s ith row.

n

gigigininiiii cacacaca

12211 A

The determinant of a matrix• The following rules are important for determinants, and

can help you sometimes to simplify calculations:– The determinant of A has the same value as the determinant of

A’.– The value of the determinant changes sign if one row (column)

is interchanged with another row (column).– If a determinant has two equal rows (columns), its value is

zero.– If a determinant has two rows (columns) with proportional

elements, its value is zero.– If all elements in a row (column) are multiplied by a constant,

the value of the determinant is multiplied by that constant.– If a determinant has a row (column) in which all elements are

zero, the value of the determinant is zero.– The value of the determinant remains unchanged if one row

(column) is added to or subtracted form another row (column). Moreover, if a row (column) is multiplied by a constant and then added to or subtracted from another row (column) the value remains unchanged.

The determinant of a matrix

• What is the determinant of:

2412

105D

123

321

842

A

10

01B

123

41C

The matrix inverse

• Let A be a square matrix. If we can find a matrix B of the same order as A such that AB=BA=I, then B is said to be the inverse of A and is symbolized A-1. A-1, if exists, can be found as follows.

• Let C be the matrix of cofactors of A (i.e., cij is the cofactor obtained from the minor |Mij|); then

• Where C’ is the transpose of C (or if one prefers, C’ is the matrix of cofactors of A’). It is immediately seen that the inverse is undefined if A is not square (since then there is no determinant |A|), and also if |A| is equal to zero.

ACA /1

The matrix inverse

• Illustration that AA-1 = A-1A = I.

2 3

4 1A

.1- 3.

.4 2.1

A

1.03.0

4.02.0

2 3

4 11AA

)1.0(2)4.0(3)3.0(2)2.0(3

)1.0(4)4.0(1)3.0(4)2.0(11AA

IAA

10

01

)2.02.1()6.06.0(

)4.04.0()2.12.0(1

The matrix inverse

• How did I get A-1?

2C

14

32C

32

C

4

32C

13

42C

1012223

41A

1.03.0

4.02.0

10

13

42

/1 AA C

2 3

4 1A Compute determinant

Now Compute C

C transpose => C’

Calculate A-1

A

The matrix inverse• Another way to calculate A-1. This way introduces you to

solving systems of equations.

.1- 3.

.4 2.1

A

2 3

4 1A

1 0

0 1

2 3

4 1

2221

12111

xx

xxIAA

10

01

2323

4141

2 3

4 1

22122111

22122111

2221

12111

xxxx

xxxx

xx

xxAA

123

041

023

141

2212

2212

2111

2111

xx

xx

xx

xx

105

;2023

141

2111

2111

2111

xx

xx

xx

205

;2123

041

2212

2212

2212

xx

xx

xx

3100

023

;3141

2111

2111

2111

xx

xx

xx

1100

123

;3041

2212

2212

2212

xx

xx

xx3.010

321

x

2.051

11 x

1.0101

22 x

4.052

12 x

The matrix inverse

• Rules for algebra with inverse matrices:– AA-1 = A-1A = I– (AB)-1 = B-1A-1

– (ABC)-1 = C-1B-1A-1

• Proof that (AB)-1 = B-1A-1.

System of equations

• In the introduction I already mentioned that the basic linear equation y=bx will be very important for multivariate methods.

• Here we will discuss how to solve systems of such linear equations.

System of equations• Illustration. Suppose we have the following set of

equations:-3=1x1+4x2

1=3x1+2x2

• The basic way to think about this problem set is finding the intersection, i.e. for which unknowns are the equations satisfied.

• This can be solved in a simple way (old style).

?.155

,2*231

413

211

21

21

xetcxx

addxx

xx

• The solution is basically the intersection of the lines represented by the equation.

• You won’t be surprised that there is a more general way to solve systems of linear equations, using matrix algebra.

System of equations

• Solution for m equations with n unknowns: m=n.

nmnmm

n

n

n x

x

x

aaa

aaa

aaa

k

k

k

2

1

21

22221

11211

2

1

Axk

Axkif AxAkA 11

xkA 1

IxkA 1

• What to do? Normally you divide by A so that you obtain a solution for x (give example: 15=3x).

• Matrix division is defined as multiplication by the inverse, so:

System of equations• Example. Suppose we have the following set of equations:

-3=1x1+4x2

1=3x1+2x2

• We already solved this one, resulting in x1=1 and x2=-1.

• The set of equations can be written as a matrix operation.

2

1

23

41

1

3

x

xAxk

System of equations

• Thus, we have to find the inverse of: A => A-1 = C’/|A|

23

41A

14

32C

4

32C

32C

2C

• We have to take the transpose of C

13

42C

System of equations

• We have to divide by |A|.

10)34()21(23

41A

13

42

10

11A

• Thus the inverse matrix is.

System of equations

2

1

1

3

13

42

10

1x

x

2

1

)11()33(

)14()32(

10

1x

x

2

1

1

1

10

10

10

1x

x

• Thus a solution for: -3=1x1+4x2 1=3x1+2x2 is found via

2

1

23

41

1

3

x

x

Axk

xkA 1

System of equationsExercise, solve: x1 + 2x2 = 0 3x1 + 7x2 = 1

1

0k

73

21A

xkA

1

2

10

20

1

0

13

271

A-1Ax = Ix = x = A-1k

• So if Ax = k solve via x = A-1k. • .... But it is not always so simple …

System of equations

• Sometimes, the requirement that m=n seems to be fulfilled, so that there should exist a solution.

• But consider the following cases.

(Row 2 = 2 x Row 1)

332

221

111

(Row 3 = Row 1 + Row 2)

1064

541

624

(Column 3 = Column 1 + Column 2), etc.

2

1

24

12

16

8

x

x

System of equations

• These situations are called linear dependence:

– Given vectors: x1, x2,…, xn-1

– Another vector xn is linearly dependent if there exists constants α1, α2,…, αn-1 such that:xn= α1x1+α2x2+ …+αn-1xn-1

• Otherwise the vector xn is linearly independent.

• In case of linear dependence; |A|= 0.• And then the inverse is not defined: A-1=C’/|A|.• And when the inverse is not defined we cannot find a

solution via: A-1k=x.

System of equations

• Generally a unique solution exists only if m=n, and |A|≠0

• When are there ‘problems’?– If m<n there are many solutions, the problem is

underdetermined. 8x1+10x2+14x3=94x1+12x2+16x3=10

– if m>n there are no solutions, the problem is overdetermined.8x1+10x2=94x1+12x2=104x1+10x2=2

System of equations

• Using the idea of linear dependency, the rank of a matrix can be introduced.

• rank(A) = number of linearly independent rows or columns.

• Given an mxn matrix, with m ≥ n, then if – |A| ≠ 0 rank(A) = n full rank, solvable– |A| = 0 rank(A) < n rank deficient

• We will get back to the issue of rank.

Overdetermined Systems

• Minimization problem: • Normal equations: (A’A)x = A’k• Solution: x = (A’A)-1A’k

– A’A must be nonsingular; i.e. |A’A|≠0– (A’A)-1A’ is called the left inverse matrix

)()( kAxkAxΦ

• Find Ax “closest” to k• Least-squares distance measure

Underdetermined Systems• Find “smallest” x that satisfies equations• Minimum norm objective

xxΦ 2

1

• Constrained minimization problem:kAx

Φ

:subject to

minx

• Solution: x = A’(AA’)-1k– AA’ must be nonsingular– A’(AA’)-1 is called the right inverse