Linear algebra primer

Instructor: Taylor Berg-KirkpatrickSlides: Sanjoy Dasgupta

Data as vectors and matrices

0 1 2 3 4 5 6 0

1

2

3

4

5

6

Matrix-vector notation

Vector x ∈ Rd :

x =

x1x2x3...xd

Matrix M ∈ Rr×d :

M =

M11 M12 · · · M1d

M21 M22 · · · M2d...

.... . .

...Mr1 Mr2 · · · Mrd

Mij = entry at row i , column j

Matrix-vector notation

Vector x ∈ Rd :

x =

x1x2x3...xd

Matrix M ∈ Rr×d :

M =

M11 M12 · · · M1d

M21 M22 · · · M2d...

.... . .

...Mr1 Mr2 · · · Mrd

Mij = entry at row i , column j

Transpose of vectors and matrices

x =

1630

has transpose xT =

M =

1 2 0 43 9 1 68 7 0 2

has transpose MT = .

• (AT )ij = Aji

• (AT )T = A

Adding and subtracting vectors and matrices

Dot product of two vectors

Dot product of vectors x , y ∈ Rd :

x · y = x1y1 + x2y2 + · · ·+ xdyd .

What is the dot product between these two vectors?

0 1 2 3 4

1

2

3

4

-1 -2 -3 -4

x

y

Dot product of two vectors

Dot product of vectors x , y ∈ Rd :

x · y = x1y1 + x2y2 + · · ·+ xdyd .

What is the dot product between these two vectors?

0 1 2 3 4

1

2

3

4

-1 -2 -3 -4

x

y

Dot products and angles

Dot product of vectors x , y ∈ Rd :x · y = x1y1 + x2y2 + · · ·+ xdyd .

Tells us the angle between x and y :

y

✓

x

cos θ =x · y‖x‖ ‖y‖

.

• x is orthogonal (at right angles) to y if and only if x · y = 0

• When x , y are unit vectors (length 1): cos θ = x · y• What is x · x?

Dot products and angles

Dot product of vectors x , y ∈ Rd :x · y = x1y1 + x2y2 + · · ·+ xdyd .

Tells us the angle between x and y :

y

✓

x

cos θ =x · y‖x‖ ‖y‖

.

• x is orthogonal (at right angles) to y if and only if x · y = 0

• When x , y are unit vectors (length 1): cos θ = x · y• What is x · x?

Dot products and angles

Dot product of vectors x , y ∈ Rd :x · y = x1y1 + x2y2 + · · ·+ xdyd .

Tells us the angle between x and y :

y

✓

x

cos θ =x · y‖x‖ ‖y‖

.

• x is orthogonal (at right angles) to y if and only if x · y = 0

• When x , y are unit vectors (length 1): cos θ = x · y

• What is x · x?

Dot products and angles

Dot product of vectors x , y ∈ Rd :x · y = x1y1 + x2y2 + · · ·+ xdyd .

Tells us the angle between x and y :

y

✓

x

cos θ =x · y‖x‖ ‖y‖

.

• x is orthogonal (at right angles) to y if and only if x · y = 0

• When x , y are unit vectors (length 1): cos θ = x · y• What is x · x?

Matrix-vector product

Product of matrix M ∈ Rr×p and vector x ∈ Rp:

Linear functions

Let M be any r × d matrix.The function x 7→ Mx sends Rd to Rr .

It is a linear function: M(x + x ′) = Mx + Mx ′.

E.g. M =

(2 0 −10 1 1

)

Write the linear function f (x1, x2) = 3x1 + 2x2 using vectornotation.

A linear function from R2 to R2 is given by M =

(1 12 2

).

As x varies, does Mx fill up all of R2?

The identity matrix

The d × d identity matrix Id sends each x ∈ Rd to itself.

Id =

1 0 0 · · · 00 1 0 · · · 00 0 1 · · · 0...

......

. . ....

0 0 0 · · · 1

Matrix-matrix product

Product of matrix A ∈ Rr×k and matrix B ∈ Rk×p:

Matrix products

If A ∈ Rr×k and B ∈ Rk×p, then AB is an r × p matrix with

(AB)ij = (dot product of ith row of A and jth column of B)

=k∑

`=1

Ai`B`j

• IkB = B and A Ik = A

• Can check: (AB)T = BTAT

• For two vectors u, v ∈ Rd , what is uT v?

Matrix products

If A ∈ Rr×k and B ∈ Rk×p, then AB is an r × p matrix with

(AB)ij = (dot product of ith row of A and jth column of B)

=k∑

`=1

Ai`B`j

• IkB = B and A Ik = A

• Can check: (AB)T = BTAT

• For two vectors u, v ∈ Rd , what is uT v?

Matrix products

If A ∈ Rr×k and B ∈ Rk×p, then AB is an r × p matrix with

(AB)ij = (dot product of ith row of A and jth column of B)

=k∑

`=1

Ai`B`j

• IkB = B and A Ik = A

• Can check: (AB)T = BTAT

• For two vectors u, v ∈ Rd , what is uT v?

Matrix products

If A ∈ Rr×k and B ∈ Rk×p, then AB is an r × p matrix with

(AB)ij = (dot product of ith row of A and jth column of B)

=k∑

`=1

Ai`B`j

• IkB = B and A Ik = A

• Can check: (AB)T = BTAT

• For two vectors u, v ∈ Rd , what is uT v?

Some special cases

For vector x ∈ Rd , what are xT x and xxT ?

Special case, generalized

For vector x ∈ Rd , we have xT x = ‖x‖2.

What about xTMx , for arbitrary d × d matrix M?

Quadratic functions

Let M be any d × d (square) matrix.For x ∈ Rd , the mapping x 7→ xTMx is a quadratic functionfrom Rd to R:

xTMx =d∑

i ,j=1

Mijxixj .

What is the quadratic function associated with M =

1 0 00 2 03 4 5

?

Write the quadratic function f (x1, x2) = x21 + 2x1x2 + 3x22 usingmatrices and vectors.

Special cases of square matrices

• Symmetric: M = MT1 2 32 4 53 5 6

,

1 2 31 2 43 4 6

• Diagonal: M = diag(m1,m2, . . . ,md)

diag(1, 4, 7) =

1 0 00 4 00 0 7

Determinant of a square matrix

Determinant of A =

(a bc d

)is |A| = ad − bc.

Example: A =

(3 11 2

)

Inverse of a square matrix

The inverse of a d × d matrix A is a d × d matrix B for whichAB = BA = Id .Notation: A−1.

Example: if A =

(1 2−2 0

)then A−1 =

(0 −1/2

1/2 1/4

). Check!

Inverse of a square matrix

The inverse of a d × d matrix A is a d × d matrix B for whichAB = BA = Id .Notation: A−1.

Example: if A =

(1 2−2 0

)then A−1 =

(0 −1/2

1/2 1/4

). Check!

Inverse of a square matrix, cont’d

The inverse of a d × d matrix A is a d × d matrix B for whichAB = BA = Id .Notation: A−1.

• Not all square matrices have an inverse

• Square matrix A is invertible if and only if |A| 6= 0

• What is the inverse of A = diag(a1, . . . , ad)?