Linear Algebra Basics

Machine Learning CS 4641

Nakul Gopalan

Georgia Tech

These slides are based on slides from Le Song and Andres Mendez-Vazquez, Chao Zhang, Mahdi Roozbahani, Rodrigo Valente

Some logistics

• Creating team.

• Office hours are started from next week.

• First quiz out this Thursday.

• First assignment out this Thursday (early release).

Outline

• Linear Algebra Basics

• Norms

• Multiplications

• Matrix Inversion

• Trace and Determinant

• Eigen Values and Eigen Vectors

• Singular Value Decomposition

• Matrix Calculus

Why Linear Algebra?

𝑛 × 𝑑𝑛 𝑑

𝑑𝑑

𝑑

𝑑

1

Example

Linear Algebra Basics

𝑛 × 𝑑 𝑑 × 𝑛

𝑑 × 𝑑

Outline

• Linear Algebra Basics

• Norms

• Multiplications

• Matrix Inversion

• Trace and Determinant

• Eigen Values and Eigen Vectors

• Singular Value Decomposition

Norms

𝑑

𝑑

𝑑

𝑑

𝑑

Norms

10

𝑑

𝑑𝑛

Vector Norm Examples

Special Matrices

𝑑 × 𝑑

𝑑

𝑑 × 𝑑

𝑑

Outline

• Linear Algebra Basics

• Norms

• Multiplications

• Matrix Inversion

• Trace and Determinant

• Eigen Values and Eigen Vectors

• Singular Value Decomposition

Multiplications

𝑛 × 𝑑 𝑑 × 𝑝

𝑛 × 𝑝 𝑑

𝑑

𝑑

𝑑 𝑛

𝑥𝑦𝑇

𝑥𝑇𝑦: 𝑥 ⊗ 𝑦

𝑥𝑦𝑇

Multiplications

T𝑥 ⊗ 𝑦 =

Inner Product Properties

Inner Product Properties

Inner Product Properties

Example

Matrix multiplication geometric meaning

Outline

• Linear Algebra Basics

• Norms

• Multiplications

• Matrix Inversion

• Trace and Determinant

• Eigen Values and Eigen Vectors

• Matrix Decomposition

Linear Independence and Matrix Rank

𝑑𝑑

𝑑

𝑑

𝑛 × 𝑑

Matrix Rank: Examples

What are the ranks for the following matrices?

Geometric meaning

Matrix Inverse

𝑑 × 𝑑

Outline

• Linear Algebra Basics

• Norms

• Multiplications

• Matrix Inversion

• Trace and Determinant

• Eigen Values and Eigen Vectors

• Singular Value Decomposition

Matrix Trace

𝑑 × 𝑑

𝑑 × 𝑑

𝑑 × 𝑑

𝑑 × 𝑑

𝑑

Matrix Determinant

𝑑

Properties of Matrix Determinant

Outline

• Linear Algebra Basics

• Norms

• Multiplications

• Matrix Inversion

• Trace and Determinant

• Eigen Values and Eigen Vectors

• Singular Value Decomposition

Eigenvalues and Eigenvectors

𝑑 × 𝑑𝑑

Computing Eigenvalues and Eigenvectors

𝑑.

𝑑 𝑑

(𝐴 − 𝜆𝐼)

(𝐴 − 𝜆𝐼) (𝐴 − 𝜆𝐼)

(𝐴 − 𝜆𝐼)

Eigenvalue Example

Slide credit: Shubham Kumbhar

Matrix Eigen Decomposition

columns

𝑑

𝑑

Outline

• Linear Algebra Basics

• Norms

• Multiplications

• Matrix Inversion

• Trace and Determinant

• Eigen Values and Eigen Vectors

• Singular Value Decomposition

Covariance matrix

Covariance matrix

Correlation matrix

Correlation matrix

Singular Value Decomposition

43

𝑋𝑛×𝑑n: instancesd: dimensionsX is a centered matrix

𝑋 = 𝑈Σ𝑉𝑇

𝑈𝑛×𝑛 → 𝑢𝑛𝑖𝑡𝑎𝑟𝑦 𝑚𝑎𝑡𝑟𝑖𝑥 → 𝑈 × 𝑈𝑇 = 𝐼

Σ𝑛×𝑑 → 𝑑𝑖𝑎𝑔𝑜𝑛𝑎𝑙 𝑚𝑎𝑡𝑟𝑖𝑥

V𝑑×𝑑 → 𝑢𝑛𝑖𝑡𝑎𝑟𝑦 𝑚𝑎𝑡𝑟𝑖𝑥 → 𝑉 × 𝑉𝑇 = 𝐼

=

ddd

d

dd

nn

n

vv

vv

uu

uu

X

.........

...

...

......

.........

000

000

00

00

00

.........

...

...

......

.........

1

11111

11

111

𝑈 Σ 𝑉𝑇

𝑑 < 𝑛

𝐶𝑑×𝑑 =𝑋𝑇𝑋

𝑛

𝑋 = 𝑈Σ𝑉𝑇

𝐶 =𝑋𝑇𝑋

𝑛

𝐶 =𝑉Σ𝑇𝑈𝑇𝑈Σ𝑉𝑇

𝑛=𝑉Σ2𝑉𝑇

𝑛

Covariance matrix:

𝐶 =𝑉Σ2𝑉𝑇

𝑛= 𝑉

Σ2

𝑛𝑉𝑇

So, we can directly calculate eigenvalue of a covariance matrix by having the singular value of matrix X directly

𝜆𝑖 =Σ𝑖2

𝑛➔ The eigenvalues of covariance matrix

According to Eigen-decomposition definition ➔𝐶𝑉 = VΛ

𝐶𝑉 = 𝑉Σ2

𝑛𝑉𝑇𝑉 = 𝑉

Σ2

𝑛

𝜆𝑖: Eigenvalue of 𝐶 or covariance matrix

Σ𝑖: Singular value of 𝑋 matrix

Geometric Meaning of SVD

Image Credit: Kevin Binz

Summary

• Linear Algebra Basics

• Norms

• Multiplications

• Matrix Inversion

• Trace and Determinant

• Eigen Values and Eigen Vectors

• Singular Value Decomposition