Upload
christiana-jefferson
View
233
Download
0
Embed Size (px)
Citation preview
2
• Simplifies notation
• Simplifies concepts
• Grounding for general linear model (GLM)
• Simplifies implementation (eg, matlab, octave)
• Optimization
Why Matrices?
3
• A matrix is a table of numbers, like a spread sheet
• Each entry is called an “element”
• There is a value in each element (no empty cells)
• Elements can be whole numbers, positive, negative, fractions, real, imaginary, symbolic variables
What is the matrix?
2 4 y
3.3i -7.4M =
M
M
4
Parts of Matrices• Size/Dimensions• Elements• Groupings of Elements• Diagonals• Triangles• Vectors
5
• Number of Rows• Number of Columns • Rows-by-Columns (RowsxCols)
Matrix Size/Dimension
2x3
2 4 y
3.3i -7.4
1x32 4 y
2 2x1
1x1
6
• Scalar is just a number or value• No dimension or size
Scalar versus Matrix
= -1
No bracketsNo dimensionNot a 1x1
= [-1]1x1
7
• Indicated by its row and column number• Scalar value
Matrix Element
2x3
(1,1) (1,2) (1,3)
(2,2) (2,3)
M =
M(2,3) = -7.4
M(i,j)
M(row,col)
Mij
2 4 y
3.3i -7.4
8
Row = Column + Offset
Diagonals
2 4 6
1 3 5
7 8 9
3x3
2 4 6
1 3 5
7 8 9
Row = Column + 0 (Main Diagonal)
Row = Column - 1 (First Upper Diagonal)
Row = Column - 2 (Second Upper Diagonal)
Row = Column - 1 (First Lower Diagonal)
Row = Column - 2 (Second Lower Diagonal)} Off-Diagonals
}
9
Triangles
2 4 6
1 3 5
7 8 93x3
Upper Triangle
Lower Triangle
10
• Special Type of Matrix • Number of Rows or Columns = 1• Often given lower-case variable names
Vector
1x3 Row Vector
2
2 4 y
2x1 Column Vector
11
A matrix consists of vectors
2 4 x
3.3i -7.4
2 4 x
3.3i -7.4
2 4 y
3.3i -7.4
Two 1x3 row vectors Three 2x1 column vectors
2x3
12
Special Matrices• Square• Diagonal• Identity• Triangular• Symmetric• Toeplitz
13
•Rows = Columns
Square Matrix
2x2
2 4
3.3i
1x1
14
• All off-diagonal elements = 0
• M(i,j)=0 if i != j
Diagonal Matrix
3x3
2 0 0
0 3 0
0 0 9
15
• Main diagonal has all 1s • All off-diagonal elements = 0• Square• Symbol “I”
Identity Matrix
3x3
1 0 0
0 1 0
0 0 1
I =
• Any matrix multiplied by the identity is itself
16
Triangular Matrices
2 4 6
0 3 5
0 0 9 3x3
Upper Triangular – all values below the main diagonal are 0
2 0 0
1 3 0
7 8 93x3
Lower Triangular – all values above the main diagonal are 0
17
Symmetric Matrix
2 4 6
4 3 5
6 5 93x3
• M(row,col) = M(col,row)
• Reflect across main diagonal
• Square
18
All elements on a diagonal are the same
Toeplitz Matrix
2 4 6
1 2 4
7 1 2 3x3
2 4
1 2
7 1 3x2
A causal, time invariant linear system is represented by an upper triangular Toeplitz matrix
19
• All elements are 1
• Any number of rows or columns
Ones Matrix
2x3
1 1 1
1 1 1
1x31 1 1
11 2x1
1x1
20
• All elements are 0
• Any number of rows or columns
Zeros Matrix
2x3
0 0 0
0 0 0
1x30 0 0
00 2x1
1x1
21
Matrix Operations• Addition/Subtraction• Vector Multiplication• Matrix Multiplication• Multiplication by scalar• Transpose• Trace• Inverse• Pseudo Inverse
22
• Element-by-Element Addition/Subtraction• Size of two matrices must be the same• C(row,col) = A(row,col) +/- B(row,col)
Matrix Addition/Subtraction
2 4 6
1 3 5A = 0 7 9
1 -2 3.3B =
2 11 15
2 1 8.3C =
2x3
2x3
2x3
C = A + B (your first matrix equation!)
23
• One Row Vector and One Column Vector• Same length, Results in 1x1• Inner Product, Dot Product, “Scalar” Product• Like a correlation• “Multiply and accumulate”
Vector Multiplication
2 4 6x =
0
71
y =
1x3
3x1
z = x.y= x*y = xi*yi
2 4 6* * *0 1 70+4+42 = 46
24
• Series of vector multiplications• Same “inner dimension”
Matrix Multiplication
2 4 6
1 3 5A =
0
1B =
2x3
3x43
4
5
7
2
6
9
8
2
3
cij=Arowi*Bcoljc11
2x4
C=A*B = c12 c13 c14
c21 c22 c23 c24
Dimensions: (2x3) *(3x4)=(2x4)
25
• Series of vector multiplications• Same “inner dimension”
Matrix Multiplication
2 4 6
1 3 5A =
0
1B =
2x3
3x43
4
5
7
2
6
9
8
2
3
c11
2x4
C=A*B = c12 c13 c14
c21 c22 c23 c24
2 4 6 013
c11=
2*0+4*1+6*3= 22
26
• Series of vector multiplications• Same “inner dimension”
Matrix Multiplication
2 4 6
1 3 5A =
0
1B =
2x3
3x43
4
5
7
2
6
9
8
2
3
22
2x4
C=A*B = c12 c13 c14
c21 c22 c23 c24
2 4 6 457
c12=
2*4+4*5+6*7= 70
27
• Series of vector multiplications• Same “inner dimension”
Matrix Multiplication
2 4 6
1 3 5A =
0
1B =
2x3
3x43
4
5
7
2
6
9
8
2
3
22
2x4
C=A*B = 70 c14
c21 c22 c23 c24
2 4 6 269
c13=
2*2+4*6+6*9= 82
c13
28
• Series of vector multiplications• Same “inner dimension”
Matrix Multiplication
2 4 6
1 3 5A =
0
1B =
2x3
3x43
4
5
7
2
6
9
8
2
3
22
2x4
C=A*B = 70 c14
c21 c22 c23 c24
2 4 6 823
c14=
2*8+4*2+6*3= 42
82
29
• Series of vector multiplications• Same “inner dimension”
Matrix Multiplication
2 4 6
1 3 5A =
0
1B =
2x3
3x43
4
5
7
2
6
9
8
2
3
22
2x4
C=A*B = 70 42
c21 c22 c23 c24
1 3 5 013
c21=
1*0+4*1+5*3= 18
82
30
• Series of vector multiplications• Same “inner dimension”
Matrix Multiplication
2 4 6
1 3 5A =
0
1B =
2x3
3x43
4
5
7
2
6
9
8
2
3
22
2x4
C=A*B = 70 42
18 c22 c23 c24
1 3 5 457
c22=
1*4+4*5+5*7= 54
82
31
• One Row Vector one Column Vector• Same length (N)• Result is NxN matrix• Special case of Matrix Multiplication
Outer Product (Vector Multiplication)
2 4 6x =
0
71
y =
1x3
3x1
z = y*x
0
71
2 4 6 1x3
3x1
0
143x3
0 0
2 4 628 42
32
• Multiply each element by scalar value
Multiplication with a scalar
2 4 6
1 3 5A = = -1
2x3
=
No bracketsNo dimensionNot a 1x1
-2 -4 -6
-1 -3 -52x3
33
• “Reflect” across diagonal• B(row,col) = A(col,row)
Matrix Transpose
2 4 6
1 3 5A =
2
4B=AT =At =A’=
2x3 3x26
1
3
5
If A=A’, then A is symmetric (must be square)
34
Sum of diagonal Elements
Trace = M(i,i)
Trace
2 4 6
1 3 5
7 8 9
M =
Trace = 2 + 3 + 9 = 14
35
Determinant
c11
2x2
C = c12
c21 c22
= c11* c22 - c12* c21
• Property of a square matrix • Complicated in general• Simple for a 2x2
36
• C*A = I, then A=C-1
• Must be square • Complicated in general• Simple for a 2x2
Matrix Inverse
c11
2x2
C = c12
c21 c22
= c11* c22 - c12* c21
c22
2x2
C-1 =-c12
-c21 c11
37
Invertibility
1.0
2x2
C = 2.0
0.5 1.0
= 1.0*1.0 - 2.0*0.5 = 1-1 = 0
1.0
2x2
C-1 =-2.0
-0.5 1.0
IMPORTANT!!!• Not all matrices are invertible• =0• “Singular”
38
Singularity and “Ill-Conditioned”
1.0
2x2
C = 2.0
0.5 1.0
= 1.0*1.0 - 2.0*0.5 = 1-1 = 0
• Column 2 = twice Column 1• Linear Dependence
Ill-Conditioned: is “close” to 0Relates to efficiency of a GLM.
39
• Can be used for non-square• Same rules on invertibility apply• Least-mean-square (LMS) for over-determined system
Pseudo-Inverse
If X has more columns than rows X+ = (X’*X)-1*X’ Check: X+*X = (X’*X)-1*X’*X=I If X has more rows than columns X+ = X’ *(X*X’)-1*
Check: X*X+ = X*X’ *(X*X’)-1=I
40
Matrix Applications
41
Sum/Length/Average of a List
length(y) = x’*x = 1*1 + 1*1 + 1*1 = 3
sum(y) = x’*y = 1*1 + 1*2 + 1*3 = 6
x =
3x1
111
1x3
x’ = [1 1 1]1x3
“Ones vector”
3x1
Average = (x’*x)-1(x’*y) [Note: GLM] = (3-1)*6 = 6/3 = 2
y =
3x1
123
42
Selective Average
length(y) = x’*x = 0*0+ 1*1 + 1*1 = 2
sum(y) = x’*y = 0*1 + 1*2 + 1*3 = 5
x =
3x1
011
1x3
x’ = [0 1 1]1x3
“Ones vector”
3x1
Average = (x’*x)-1(x’*y) = (2-1)*5 = 5/2 = 2.5
y =
3x1
123
43
Sum of the Squares of a List
y’ = [1 2 3]1x3
SS(y) = y’*y = 1*1 + 2*2 + 3*3 = 14
y =
3x1
123
SS =y2)
44
Variance of a List
Residual e = y – *x2 = sqrt(e’*e)/DOFDOF = x’*x - 1 = #RowsX-#ColsX
y =
3x1
123
x =
3x1
111
2 =(y-)2
(N-1)
45
• Vector product is 0• Property of two vectors• Uncorrelated/Independence - statistical property
Orthogonality
1 -1 5x =
2
.23
y =
1x3
3x1
z = x*y = xi*yi
1 * 2 +-1 * 3 + 5 * .2 = [0]
1x1
46
Fitting a Line
HRF AmplitudeIQ, Height, Weight
Independent Variable
DependentVariable,Measurement
x1 x2
y2
y1
Subject 1
Subject 2
Thickness
Age
47
Linear ModelIntercept: b
Slope: m
Age
x1 x2
y2
y1
System of Linear Equationsy1 = 1*b + x1*my2 = 1*b + x2*mTwo Eqns, Two Unknowns
“Intercept” = “Offset”
m= (y2-y1) (x2-x1)
b= (x2*y1-x1*y2) (x2-x1)
1. Solve 1st Eq for b2. Substitute into 2nd Eq3. Solve 2nd equation for m4. Substitute m back into Eq for b
48
Matrix Formulation/GLM
Intercept: b
Slope: m
Age
x1 x2
y2
y1
y1 = 1*b + x1*my2 = 1*b + x2*m
- parameter vectorX - design matrix
y1y2
1 x11 x2
bm= *
y = X*2x2
2x12x1
49
Matrix Formulation/GLMIntercept: b
Slope: m
Age
x1 x2
y2
y1
y = X*
y1y2
1 x11 x2
bm= *
1X =
x1
1 x2
•# of Cols of X = # Parameters•Each parameter has a column•Each column means something•Parameter meaning from column
50
GLM SolutionIntercept: b
Slope: m
Age
x1 x2
y2
y1
y = X*ultiply both sides by X-1
=X-1*y
y1y2
1 x11 x2
bm= *
1X =
x1
1 x2
x2X-1 =
-x1
-1 11
= x2-x1Non-invertible if x1=x2Ill-conditioned if x1 near x2 -- Sensitive to noise
51
Non-invertibility
Intercept: b
Slope: m
Age
x1 x2
y2
y1
y1y2
1 x11 x2
bm= *
1X =
.1
1 .1
y = X*1.2
y = 1.2
1 = 2
.2 = 10
1.2y = 1.2
x1=x2=0.1 Want to find but …
is not unique!
52
More than Two Pointsy1 = 1*b + x1*my2 = 1*b + x2*my3 = 1*b + x3*m
Intercept: b
Slope: m
y = X*ultiply both sides by X+
=X+y=(X’X)-1X’y• Three Eqns, Two Unknowns• Over-determined• Still two columns in X, same meaning• X is not square (use pseudo-inv X+; LMS)• Same invertibility concerns• Measure noise (DOF != 0)
53
Matrix Decomposition/Factorization
Break up a single matrix into several matrices that are multiplied together:• Eigen System• Singular Value Decomposition (SVD)
• Principle Component Analysis (PCA)• Cholesky Decomposition - “square root” of a matrix
54
Eigen Decomposition
A = Q**Q-1, A*v = *v
A – square matrixv – eigenvector- eigenvalueQ – matrix of eigenvectors – diagonal matrix of eigenvalues
55