Linear Algebra Basics
Machine Learning CS 4641
Nakul Gopalan
Georgia Tech
These slides are based on slides from Le Song and Andres Mendez-Vazquez, Chao Zhang, Mahdi Roozbahani, Rodrigo Valente
Some logistics
• Creating team.
• Office hours are started from next week.
• First quiz out this Thursday.
• First assignment out this Thursday (early release).
Outline
• Linear Algebra Basics
• Norms
• Multiplications
• Matrix Inversion
• Trace and Determinant
• Eigen Values and Eigen Vectors
• Singular Value Decomposition
• Matrix Calculus
Why Linear Algebra?
𝑛 × 𝑑𝑛 𝑑
𝑑𝑑
𝑑
𝑑
1
Example
Linear Algebra Basics
𝑛 × 𝑑 𝑑 × 𝑛
𝑑 × 𝑑
Outline
• Linear Algebra Basics
• Norms
• Multiplications
• Matrix Inversion
• Trace and Determinant
• Eigen Values and Eigen Vectors
• Singular Value Decomposition
Norms
𝑑
𝑑
𝑑
𝑑
𝑑
Norms
10
𝑑
𝑑𝑛
Vector Norm Examples
Special Matrices
𝑑 × 𝑑
𝑑
𝑑 × 𝑑
𝑑
Outline
• Linear Algebra Basics
• Norms
• Multiplications
• Matrix Inversion
• Trace and Determinant
• Eigen Values and Eigen Vectors
• Singular Value Decomposition
Multiplications
𝑛 × 𝑑 𝑑 × 𝑝
𝑛 × 𝑝 𝑑
𝑑
𝑑
𝑑 𝑛
𝑥𝑦𝑇
𝑥𝑇𝑦: 𝑥 ⊗ 𝑦
𝑥𝑦𝑇
Multiplications
T𝑥 ⊗ 𝑦 =
Inner Product Properties
Inner Product Properties
Inner Product Properties
Example
Matrix multiplication geometric meaning
Outline
• Linear Algebra Basics
• Norms
• Multiplications
• Matrix Inversion
• Trace and Determinant
• Eigen Values and Eigen Vectors
• Matrix Decomposition
Linear Independence and Matrix Rank
𝑑𝑑
𝑑
𝑑
𝑛 × 𝑑
Matrix Rank: Examples
What are the ranks for the following matrices?
Geometric meaning
Matrix Inverse
𝑑 × 𝑑
Outline
• Linear Algebra Basics
• Norms
• Multiplications
• Matrix Inversion
• Trace and Determinant
• Eigen Values and Eigen Vectors
• Singular Value Decomposition
Matrix Trace
𝑑 × 𝑑
𝑑 × 𝑑
𝑑 × 𝑑
𝑑 × 𝑑
𝑑
Matrix Determinant
𝑑
Properties of Matrix Determinant
Outline
• Linear Algebra Basics
• Norms
• Multiplications
• Matrix Inversion
• Trace and Determinant
• Eigen Values and Eigen Vectors
• Singular Value Decomposition
Eigenvalues and Eigenvectors
𝑑 × 𝑑𝑑
Computing Eigenvalues and Eigenvectors
𝑑.
𝑑 𝑑
(𝐴 − 𝜆𝐼)
(𝐴 − 𝜆𝐼) (𝐴 − 𝜆𝐼)
(𝐴 − 𝜆𝐼)
Eigenvalue Example
Slide credit: Shubham Kumbhar
Matrix Eigen Decomposition
columns
𝑑
𝑑
Outline
• Linear Algebra Basics
• Norms
• Multiplications
• Matrix Inversion
• Trace and Determinant
• Eigen Values and Eigen Vectors
• Singular Value Decomposition
Covariance matrix
Covariance matrix
Correlation matrix
Correlation matrix
Singular Value Decomposition
43
𝑋𝑛×𝑑n: instancesd: dimensionsX is a centered matrix
𝑋 = 𝑈Σ𝑉𝑇
𝑈𝑛×𝑛 → 𝑢𝑛𝑖𝑡𝑎𝑟𝑦 𝑚𝑎𝑡𝑟𝑖𝑥 → 𝑈 × 𝑈𝑇 = 𝐼
Σ𝑛×𝑑 → 𝑑𝑖𝑎𝑔𝑜𝑛𝑎𝑙 𝑚𝑎𝑡𝑟𝑖𝑥
V𝑑×𝑑 → 𝑢𝑛𝑖𝑡𝑎𝑟𝑦 𝑚𝑎𝑡𝑟𝑖𝑥 → 𝑉 × 𝑉𝑇 = 𝐼
=
ddd
d
dd
nn
n
vv
vv
uu
uu
X
.........
...
...
......
.........
000
000
00
00
00
.........
...
...
......
.........
1
11111
11
111
𝑈 Σ 𝑉𝑇
𝑑 < 𝑛
𝐶𝑑×𝑑 =𝑋𝑇𝑋
𝑛
𝑋 = 𝑈Σ𝑉𝑇
𝐶 =𝑋𝑇𝑋
𝑛
𝐶 =𝑉Σ𝑇𝑈𝑇𝑈Σ𝑉𝑇
𝑛=𝑉Σ2𝑉𝑇
𝑛
Covariance matrix:
𝐶 =𝑉Σ2𝑉𝑇
𝑛= 𝑉
Σ2
𝑛𝑉𝑇
So, we can directly calculate eigenvalue of a covariance matrix by having the singular value of matrix X directly
𝜆𝑖 =Σ𝑖2
𝑛➔ The eigenvalues of covariance matrix
According to Eigen-decomposition definition ➔𝐶𝑉 = VΛ
𝐶𝑉 = 𝑉Σ2
𝑛𝑉𝑇𝑉 = 𝑉
Σ2
𝑛
𝜆𝑖: Eigenvalue of 𝐶 or covariance matrix
Σ𝑖: Singular value of 𝑋 matrix
Geometric Meaning of SVD
Image Credit: Kevin Binz
Summary
• Linear Algebra Basics
• Norms
• Multiplications
• Matrix Inversion
• Trace and Determinant
• Eigen Values and Eigen Vectors
• Singular Value Decomposition
Recommended
